[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:17:59 dist-test-slave-lsvg krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:17:59 dist-test-slave-lsvg krb5kdc[399](info): set up 2 sockets
Sep 27 11:17:59 dist-test-slave-lsvg krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 11:18:00.353977 392 mini_kdc.cc:121] Time spent starting KDC: real 2.186s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:00 dist-test-slave-lsvg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:18:00.472674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45725
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:00.554502 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:00.554705 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:00.554726 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:00.555845 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:00.555884 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:00.555899 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:00.555930 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:00.555943 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:00.557402 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:00.557638 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:00.557895 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:00.562927 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:00.562950 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:00.562958 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:00.563084 412 server_base.cc:1034] running on GCE node
I20260927 11:18:00.563385 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:00.563732 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:00.564903 412 hybrid_clock.cc:648] HybridClock initialized: now 1790507880564863 us; error 66 us; skew 500 ppm
Sep 27 11:18:00 dist-test-slave-lsvg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:00.570794 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:18:00.571280 412 webserver.cc:460] Webserver started at http://127.0.98.62:40615/ using document root <none> and password file <none>
I20260927 11:18:00.571507 412 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:00.571574 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:00.571738 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:00.572858 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "04a57ac25725494c8536b42e8f1b4df8"
format_stamp: "Formatted at 2026-09-27 11:18:00 on dist-test-slave-lsvg"
server_key: "0328ffaead983ad2e8d8ab7568cc32d0"
server_key_iv: "57ec25520ff0bae98d2677b188c11235"
server_key_version: "encryptionkey@0"
I20260927 11:18:00.573246 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "04a57ac25725494c8536b42e8f1b4df8"
format_stamp: "Formatted at 2026-09-27 11:18:00 on dist-test-slave-lsvg"
server_key: "0328ffaead983ad2e8d8ab7568cc32d0"
server_key_iv: "57ec25520ff0bae98d2677b188c11235"
server_key_version: "encryptionkey@0"
I20260927 11:18:00.574633 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:00.575636 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:00.575865 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:00.575973 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "04a57ac25725494c8536b42e8f1b4df8"
format_stamp: "Formatted at 2026-09-27 11:18:00 on dist-test-slave-lsvg"
server_key: "0328ffaead983ad2e8d8ab7568cc32d0"
server_key_iv: "57ec25520ff0bae98d2677b188c11235"
server_key_version: "encryptionkey@0"
I20260927 11:18:00.576046 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:00.585283 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:00.587643 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:00.587821 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:00.591672 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45725
I20260927 11:18:00.591703 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45725 every 8 connection(s)
I20260927 11:18:00.592059 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:00.592941 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:00.595278 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8: Bootstrap starting.
I20260927 11:18:00.595882 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:00.596192 480 log.cc:824] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:00.596918 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8: No bootstrap required, opened a new log
I20260927 11:18:00.597406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 412
I20260927 11:18:00.597522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:00.598167 392 external_mini_cluster.cc:1428] Setting key 2902d58487b210f8c2f2815f42e618fa
I20260927 11:18:00.598753 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a57ac25725494c8536b42e8f1b4df8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45725 } }
I20260927 11:18:00.598939 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:00.598980 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a57ac25725494c8536b42e8f1b4df8, State: Initialized, Role: FOLLOWER
I20260927 11:18:00.599124 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a57ac25725494c8536b42e8f1b4df8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45725 } }
I20260927 11:18:00.599222 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:00.599267 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:00.599339 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:00.600265 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a57ac25725494c8536b42e8f1b4df8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45725 } }
I20260927 11:18:00.600356 480 leader_election.cc:302] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04a57ac25725494c8536b42e8f1b4df8; no voters:
I20260927 11:18:00.600539 480 leader_election.cc:288] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:00.600580 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:00.600761 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [term 1 LEADER]: Becoming Leader. State: Replica: 04a57ac25725494c8536b42e8f1b4df8, State: Running, Role: LEADER
I20260927 11:18:00.600840 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:00.600945 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a57ac25725494c8536b42e8f1b4df8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45725 } }
I20260927 11:18:00.601357 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04a57ac25725494c8536b42e8f1b4df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a57ac25725494c8536b42e8f1b4df8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45725 } } }
I20260927 11:18:00.601432 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:00.601575 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04a57ac25725494c8536b42e8f1b4df8. Latest consensus state: current_term: 1 leader_uuid: "04a57ac25725494c8536b42e8f1b4df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a57ac25725494c8536b42e8f1b4df8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45725 } } }
I20260927 11:18:00.601624 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:00.602046 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:00.602640 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:00.604704 494 catalog_manager.cc:1347] Generated new cluster ID: 5e54ab5d28a341f3afbedc0823afe899
I20260927 11:18:00.604758 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:18:00 dist-test-slave-lsvg krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:00.614336 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:00.599124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.0.98.62:45725)
0927 11:18:00.599439 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:00.599449 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:00.599623 (+ 174us) server_negotiation.cc:407] Connection header received
0927 11:18:00.601848 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:00.601857 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:00.601913 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:00.602030 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:00.604362 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:00.605496 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:00.606551 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:00.606982 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:00.607115 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:00.610551 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:00.610555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:00.610555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:00.610566 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:00.612265 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:00.612733 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:00.612736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:00.612736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:00.612842 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:00.613139 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:00.613139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:00.613140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:00.613389 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:00.613579 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:00.613958 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 11:18:00.614102 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":136,"threads_started":1}
I20260927 11:18:00.614431 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:00.615074 494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:00.622666 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04a57ac25725494c8536b42e8f1b4df8: Generated new TSK 0
I20260927 11:18:00.622875 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:18:00.652693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45725
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:00.754046 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:00.754262 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:00.754287 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:00.755427 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:00.755468 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:00.755496 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:00.756917 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45725
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:00.757195 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:00.757457 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:00.762614 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:00.762634 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:00.762589 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:00.762877 506 server_base.cc:1034] running on GCE node
I20260927 11:18:00.763074 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:00.763314 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:00.764483 506 hybrid_clock.cc:648] HybridClock initialized: now 1790507880764476 us; error 39 us; skew 500 ppm
Sep 27 11:18:00 dist-test-slave-lsvg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:00.767190 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:18:00.767690 506 webserver.cc:460] Webserver started at http://127.0.98.1:41121/ using document root <none> and password file <none>
I20260927 11:18:00.767923 506 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:00.767978 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:00.768103 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:00.769490 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "73e6e1489de5475db4de5463d7037e7c"
format_stamp: "Formatted at 2026-09-27 11:18:00 on dist-test-slave-lsvg"
server_key: "1d3f48dbcf4d1cee5ca010b59e177909"
server_key_iv: "6cfce6d68d3901a4731204b446db9a22"
server_key_version: "encryptionkey@0"
I20260927 11:18:00.769935 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "73e6e1489de5475db4de5463d7037e7c"
format_stamp: "Formatted at 2026-09-27 11:18:00 on dist-test-slave-lsvg"
server_key: "1d3f48dbcf4d1cee5ca010b59e177909"
server_key_iv: "6cfce6d68d3901a4731204b446db9a22"
server_key_version: "encryptionkey@0"
I20260927 11:18:00.771601 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:18:00.772420 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:00.772615 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:00.772709 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "73e6e1489de5475db4de5463d7037e7c"
format_stamp: "Formatted at 2026-09-27 11:18:00 on dist-test-slave-lsvg"
server_key: "1d3f48dbcf4d1cee5ca010b59e177909"
server_key_iv: "6cfce6d68d3901a4731204b446db9a22"
server_key_version: "encryptionkey@0"
I20260927 11:18:00.772776 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:00.791733 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:00.794056 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:00.794178 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:00.794370 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:00.794711 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:00.794747 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:00.794770 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:00.794821 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:00.800953 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33841
I20260927 11:18:00.801014 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33841 every 8 connection(s)
I20260927 11:18:00.801446 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:18:00 dist-test-slave-lsvg krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:00.808144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 506
I20260927 11:18:00.808252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:00.808442 392 external_mini_cluster.cc:1428] Setting key 371562f1e56736c4768a3a9fb43d5323
I20260927 11:18:00.811142 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:00.802166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43195 (local address 127.0.98.62:45725)
0927 11:18:00.802230 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:00.802233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:00.802521 (+ 288us) server_negotiation.cc:407] Connection header received
0927 11:18:00.803033 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:00.803034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:00.803046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:00.803101 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:00.804056 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:00.804582 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:00.805642 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:00.805839 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:00.805895 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:00.808071 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:00.808073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:00.808073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:00.808081 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:00.809701 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:00.810143 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:00.810145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:00.810145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:00.810208 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:00.810476 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:00.810477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:00.810477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:00.810592 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:00.810667 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:00.810962 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 11:18:00.811052 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 11:18:00.811355 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:00.802167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45725 (local address 127.0.98.1:43195)
0927 11:18:00.802351 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:00.802395 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:00.802933 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:00.803161 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:00.803164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:00.803752 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:00.803949 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:00.803954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:00.804664 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:00.804664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:00.805567 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:00.805570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:00.805914 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:00.805915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:00.806017 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:00.806496 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:00.806505 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:00.807947 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:00.809817 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:00.809824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:00.809827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:00.810063 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:00.810281 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:00.810282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:00.810283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:00.810403 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:00.810659 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:00.810666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:00.810819 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 11:18:00.810928 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 11:18:00.811095 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260927 11:18:00.811831 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45725
I20260927 11:18:00.811940 635 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:00.812137 635 heartbeater.cc:511] Master 127.0.98.62:45725 requested a full tablet report, sending...
I20260927 11:18:00.812753 444 ts_manager.cc:194] Registered new tserver with Master: 73e6e1489de5475db4de5463d7037e7c (127.0.98.1:33841)
I20260927 11:18:00.813467 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43195
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:18:00.834241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45725
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:00.914027 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:00.914238 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:00.914270 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:00.915452 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:00.915498 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:00.915526 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:00.917043 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45725
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:00.917318 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:00.917542 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:00.922683 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:00.922730 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:00.922878 640 server_base.cc:1034] running on GCE node
W20260927 11:18:00.922730 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:00.923135 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:00.923401 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:00.924562 640 hybrid_clock.cc:648] HybridClock initialized: now 1790507880924568 us; error 47 us; skew 500 ppm
Sep 27 11:18:00 dist-test-slave-lsvg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:00.927066 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:18:00.927529 640 webserver.cc:460] Webserver started at http://127.0.98.2:41165/ using document root <none> and password file <none>
I20260927 11:18:00.927762 640 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:00.927831 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:00.927992 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:00.929116 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "04f3cff3b30a46a5b22fac5fc3810769"
format_stamp: "Formatted at 2026-09-27 11:18:00 on dist-test-slave-lsvg"
server_key: "cc1c4a78122f2629e2e8a44cdec11e1f"
server_key_iv: "43dd8e2e9440b592d8d11e63c657e608"
server_key_version: "encryptionkey@0"
I20260927 11:18:00.929497 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "04f3cff3b30a46a5b22fac5fc3810769"
format_stamp: "Formatted at 2026-09-27 11:18:00 on dist-test-slave-lsvg"
server_key: "cc1c4a78122f2629e2e8a44cdec11e1f"
server_key_iv: "43dd8e2e9440b592d8d11e63c657e608"
server_key_version: "encryptionkey@0"
I20260927 11:18:00.930845 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:00.931594 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:00.931792 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:00.931886 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "04f3cff3b30a46a5b22fac5fc3810769"
format_stamp: "Formatted at 2026-09-27 11:18:00 on dist-test-slave-lsvg"
server_key: "cc1c4a78122f2629e2e8a44cdec11e1f"
server_key_iv: "43dd8e2e9440b592d8d11e63c657e608"
server_key_version: "encryptionkey@0"
I20260927 11:18:00.931970 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:00.949455 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:00.951439 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:00.951598 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:00.951864 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:00.952179 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:00.952230 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:00.952267 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:00.952296 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:00.958287 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44149
I20260927 11:18:00.958328 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44149 every 8 connection(s)
I20260927 11:18:00.958636 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:18:00.959370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 640
I20260927 11:18:00.959481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:00.959666 392 external_mini_cluster.cc:1428] Setting key e636605238050c03c8c28e66f4eb3435
Sep 27 11:18:00 dist-test-slave-lsvg krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 11:18:00.970228 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:00.959351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35303 (local address 127.0.98.62:45725)
0927 11:18:00.959397 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:00.959400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:00.960031 (+ 631us) server_negotiation.cc:407] Connection header received
0927 11:18:00.960658 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:00.960659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:00.960672 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:00.960744 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:00.961958 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:00.962469 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:00.964006 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:00.964200 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:00.964257 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:00.966934 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:00.966936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:00.966936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:00.966952 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:00.968621 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:00.969149 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:00.969150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:00.969151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:00.969221 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:00.969535 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:00.969536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:00.969536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:00.969669 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:00.969730 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:00.970074 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 11:18:00.970152 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:18:00.970468 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:00.959357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45725 (local address 127.0.98.2:35303)
0927 11:18:00.959882 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:00.959932 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:00.960549 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:00.960835 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:00.960838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:00.961556 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:00.961854 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:00.961861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:00.962562 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:00.962563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:00.963924 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:00.963927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:00.964291 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:00.964292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:00.964444 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:00.964974 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:00.964986 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:00.966803 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:00.968713 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:00.968720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:00.968722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:00.968964 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:00.969305 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:00.969306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:00.969306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:00.969426 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:00.969746 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:00.969752 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:00.969885 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 11:18:00.970027 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 11:18:00.970235 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":84,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 11:18:00.971053 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45725
I20260927 11:18:00.971163 769 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:00.971379 769 heartbeater.cc:511] Master 127.0.98.62:45725 requested a full tablet report, sending...
I20260927 11:18:00.971804 444 ts_manager.cc:194] Registered new tserver with Master: 04f3cff3b30a46a5b22fac5fc3810769 (127.0.98.2:44149)
I20260927 11:18:00.972313 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:18:00.988190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45725
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:01.074195 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:01.074398 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:01.074426 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:01.075657 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:01.075709 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:01.075759 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:01.077330 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45725
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:01.077571 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:01.077813 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:01.082840 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:01.082914 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:01.082996 774 server_base.cc:1034] running on GCE node
W20260927 11:18:01.083151 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:01.083382 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:01.083629 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:01.084820 774 hybrid_clock.cc:648] HybridClock initialized: now 1790507881084811 us; error 49 us; skew 500 ppm
Sep 27 11:18:01 dist-test-slave-lsvg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:01.087477 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:18:01.087972 774 webserver.cc:460] Webserver started at http://127.0.98.3:33261/ using document root <none> and password file <none>
I20260927 11:18:01.088223 774 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:01.088292 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:01.088446 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:01.089967 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "d94db9690e4146b5862315ac4fbda959"
format_stamp: "Formatted at 2026-09-27 11:18:01 on dist-test-slave-lsvg"
server_key: "74847b32116c44aae29bee007671388b"
server_key_iv: "455caf6a384b9d6282dd8aaa5b16533e"
server_key_version: "encryptionkey@0"
I20260927 11:18:01.090358 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d94db9690e4146b5862315ac4fbda959"
format_stamp: "Formatted at 2026-09-27 11:18:01 on dist-test-slave-lsvg"
server_key: "74847b32116c44aae29bee007671388b"
server_key_iv: "455caf6a384b9d6282dd8aaa5b16533e"
server_key_version: "encryptionkey@0"
I20260927 11:18:01.091678 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:18:01.092402 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:01.092573 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:01.092664 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "d94db9690e4146b5862315ac4fbda959"
format_stamp: "Formatted at 2026-09-27 11:18:01 on dist-test-slave-lsvg"
server_key: "74847b32116c44aae29bee007671388b"
server_key_iv: "455caf6a384b9d6282dd8aaa5b16533e"
server_key_version: "encryptionkey@0"
I20260927 11:18:01.092747 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:01.106643 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:01.109018 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:01.109265 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:01.109511 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:01.109829 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:01.109884 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:01.109925 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:01.109952 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:01.116693 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40487
I20260927 11:18:01.116791 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40487 every 8 connection(s)
I20260927 11:18:01.117154 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 11:18:01 dist-test-slave-lsvg krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:01.123530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 774
I20260927 11:18:01.123626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:01.123819 392 external_mini_cluster.cc:1428] Setting key 5eae51183b466e80c8b1c42a5c5b12a1
I20260927 11:18:01.126636 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:01.118004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42909 (local address 127.0.98.62:45725)
0927 11:18:01.118081 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:01.118083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:01.118289 (+ 206us) server_negotiation.cc:407] Connection header received
0927 11:18:01.118752 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:01.118754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:01.118766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:01.118831 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:01.119759 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:01.120276 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:01.121348 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:01.121535 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:01.121597 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:01.123769 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:01.123771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:01.123771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:01.123780 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:01.125466 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:01.125830 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:01.125831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:01.125831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:01.125890 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:01.126097 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:01.126098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:01.126098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:01.126231 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:01.126283 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:01.126517 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 11:18:01.126575 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 11:18:01.126870 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:01.117967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45725 (local address 127.0.98.3:42909)
0927 11:18:01.118138 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:01.118182 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:01.118657 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:01.118888 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:01.118892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:01.119458 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:01.119653 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:01.119658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:01.120359 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:01.120359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:01.121270 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:01.121272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:01.121609 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:01.121610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:01.121721 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:01.122230 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:01.122238 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:01.123652 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:01.125534 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:01.125539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:01.125541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:01.125764 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:01.125949 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:01.125949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:01.125950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:01.126043 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:01.126287 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:01.126290 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:01.126408 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 11:18:01.126491 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 11:18:01.126607 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260927 11:18:01.127318 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45725
I20260927 11:18:01.127395 903 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:01.127581 903 heartbeater.cc:511] Master 127.0.98.62:45725 requested a full tablet report, sending...
I20260927 11:18:01.127966 444 ts_manager.cc:194] Registered new tserver with Master: d94db9690e4146b5862315ac4fbda959 (127.0.98.3:40487)
I20260927 11:18:01.128341 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42909
I20260927 11:18:01.135936 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 11:18:01 dist-test-slave-lsvg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507881, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 11:18:01 dist-test-slave-lsvg krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507881, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:18:01.160142 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:01.151452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.0.98.1:33841)
0927 11:18:01.151620 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:01.151625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:01.151635 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:01.151662 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:01.151664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:01.151678 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:01.151750 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:01.152448 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:01.153295 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:01.154026 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:01.154376 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:01.154435 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:01.156868 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:01.156876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:01.156876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:01.156883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:01.158568 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:01.158951 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:01.158954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:01.158954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:01.159022 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:01.159286 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:01.159286 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:01.159287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:01.159426 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:01.159537 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:01.159899 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 11:18:01.159986 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
W20260927 11:18:01.160553 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47104
Sep 27 11:18:01 dist-test-slave-lsvg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 11:18:01 dist-test-slave-lsvg krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:18:01.183194 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:01.175614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.0.98.1:33841)
0927 11:18:01.175708 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:01.175710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:01.175758 (+ 48us) server_negotiation.cc:407] Connection header received
0927 11:18:01.175870 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:01.175871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:01.175887 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:01.175957 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:01.176614 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:01.177169 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:01.177975 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:01.178182 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:01.178256 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:01.180648 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:01.180650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:01.180650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:01.180659 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:01.182074 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:01.182433 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:01.182434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:01.182435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:01.182491 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:01.182688 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:01.182689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:01.182689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:01.182793 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:01.182920 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:01.183005 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 11:18:01.183087 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 11:18:01.183728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 506
I20260927 11:18:01.188290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 640
I20260927 11:18:01.194016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 774
I20260927 11:18:01.198093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 412
2026-09-27T11:18:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3057 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:01 dist-test-slave-lsvg krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:01 dist-test-slave-lsvg krb5kdc[917](info): set up 2 sockets
Sep 27 11:18:01 dist-test-slave-lsvg krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 11:18:02.240561 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:02 dist-test-slave-lsvg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:18:02.363603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37343
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:02.442580 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:02.442783 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:02.442806 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:02.443881 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:02.443917 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:02.443933 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:02.443944 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:02.443956 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:02.445374 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:02.445564 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:02.445741 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:02.450421 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:02.450400 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:02.450500 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:02.450595 930 server_base.cc:1034] running on GCE node
I20260927 11:18:02.450899 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:02.451228 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:02.452376 930 hybrid_clock.cc:648] HybridClock initialized: now 1790507882452362 us; error 27 us; skew 500 ppm
Sep 27 11:18:02 dist-test-slave-lsvg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:02.454838 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:18:02.455281 930 webserver.cc:460] Webserver started at http://127.0.98.62:39203/ using document root <none> and password file <none>
I20260927 11:18:02.455628 930 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:02.455718 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:02.455850 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:02.456866 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "1f379c4f1d7e4118a414c6bdf7ecc743"
format_stamp: "Formatted at 2026-09-27 11:18:02 on dist-test-slave-lsvg"
server_key: "f3b1459a6a2399d6127ad44efb5deea4"
server_key_iv: "b51e298a3a6e1994ac3f92bf66d76f42"
server_key_version: "encryptionkey@0"
I20260927 11:18:02.457249 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f379c4f1d7e4118a414c6bdf7ecc743"
format_stamp: "Formatted at 2026-09-27 11:18:02 on dist-test-slave-lsvg"
server_key: "f3b1459a6a2399d6127ad44efb5deea4"
server_key_iv: "b51e298a3a6e1994ac3f92bf66d76f42"
server_key_version: "encryptionkey@0"
I20260927 11:18:02.458657 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:02.459475 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:02.459658 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:02.459762 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "1f379c4f1d7e4118a414c6bdf7ecc743"
format_stamp: "Formatted at 2026-09-27 11:18:02 on dist-test-slave-lsvg"
server_key: "f3b1459a6a2399d6127ad44efb5deea4"
server_key_iv: "b51e298a3a6e1994ac3f92bf66d76f42"
server_key_version: "encryptionkey@0"
I20260927 11:18:02.459849 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:02.472169 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:02.474429 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:02.474607 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:02.478844 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37343
I20260927 11:18:02.478871 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37343 every 8 connection(s)
I20260927 11:18:02.479255 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:02.480149 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:02.482425 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743: Bootstrap starting.
I20260927 11:18:02.483012 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:02.483309 998 log.cc:824] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:02.483985 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743: No bootstrap required, opened a new log
I20260927 11:18:02.485567 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f379c4f1d7e4118a414c6bdf7ecc743" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37343 } }
I20260927 11:18:02.485699 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:02.485723 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f379c4f1d7e4118a414c6bdf7ecc743, State: Initialized, Role: FOLLOWER
I20260927 11:18:02.485874 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f379c4f1d7e4118a414c6bdf7ecc743" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37343 } }
I20260927 11:18:02.485952 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:02.485975 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:02.486003 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:02.486529 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f379c4f1d7e4118a414c6bdf7ecc743" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37343 } }
I20260927 11:18:02.486611 998 leader_election.cc:302] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f379c4f1d7e4118a414c6bdf7ecc743; no voters:
I20260927 11:18:02.486786 998 leader_election.cc:288] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:02.486833 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:02.487048 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:02.487234 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [term 1 LEADER]: Becoming Leader. State: Replica: 1f379c4f1d7e4118a414c6bdf7ecc743, State: Running, Role: LEADER
I20260927 11:18:02.487344 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f379c4f1d7e4118a414c6bdf7ecc743" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37343 } }
I20260927 11:18:02.487735 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f379c4f1d7e4118a414c6bdf7ecc743. Latest consensus state: current_term: 1 leader_uuid: "1f379c4f1d7e4118a414c6bdf7ecc743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f379c4f1d7e4118a414c6bdf7ecc743" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37343 } } }
I20260927 11:18:02.487824 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:02.488173 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f379c4f1d7e4118a414c6bdf7ecc743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f379c4f1d7e4118a414c6bdf7ecc743" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37343 } } }
I20260927 11:18:02.488255 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:02.488251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 930
I20260927 11:18:02.488332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:02.488505 392 external_mini_cluster.cc:1428] Setting key d99b6fb04009b3fc3850fe64d177c48e
I20260927 11:18:02.488035 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:02.489223 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:02.490672 1010 catalog_manager.cc:1347] Generated new cluster ID: 149fa154c8844613a9677cf713f14a93
I20260927 11:18:02.490724 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:18:02 dist-test-slave-lsvg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:02.503132 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:02.489071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.0.98.62:37343)
0927 11:18:02.493405 (+ 4334us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:02.493419 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:02.493444 (+ 25us) server_negotiation.cc:407] Connection header received
0927 11:18:02.493562 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:02.493569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:02.493628 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:02.493809 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:02.494640 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:02.495591 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:02.496420 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:02.496836 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:02.496967 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:02.499479 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:02.499484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:02.499484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:02.499502 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:02.501457 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:02.501877 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:02.501880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:02.501881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:02.501974 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:02.502189 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:02.502190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:02.502191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:02.502439 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:02.502609 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:02.502714 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 11:18:02.502956 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4278,"thread_start_us":73,"threads_started":1}
I20260927 11:18:02.509729 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:02.510489 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:02.518999 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f379c4f1d7e4118a414c6bdf7ecc743: Generated new TSK 0
I20260927 11:18:02.519177 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:18:02.550405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37343
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:02.648927 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:02.649211 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:02.649243 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:02.650463 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:02.650506 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:02.650535 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:02.652001 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:02.652243 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:02.652498 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:02.657712 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:02.657823 1024 server_base.cc:1034] running on GCE node
W20260927 11:18:02.657722 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:02.657722 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:02.658159 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:02.658458 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:02.659643 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790507882659633 us; error 38 us; skew 500 ppm
Sep 27 11:18:02 dist-test-slave-lsvg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:02.662483 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:18:02.663030 1024 webserver.cc:460] Webserver started at http://127.0.98.1:42217/ using document root <none> and password file <none>
I20260927 11:18:02.663281 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:02.663340 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:02.663509 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:02.664907 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "b5bc97cb357f4ec588b2cf796085cda1"
format_stamp: "Formatted at 2026-09-27 11:18:02 on dist-test-slave-lsvg"
server_key: "b5df56a9a3799870fe6078ae4d8fc0b8"
server_key_iv: "b27fa28331cc0d09951fd677a33d25e0"
server_key_version: "encryptionkey@0"
I20260927 11:18:02.665385 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b5bc97cb357f4ec588b2cf796085cda1"
format_stamp: "Formatted at 2026-09-27 11:18:02 on dist-test-slave-lsvg"
server_key: "b5df56a9a3799870fe6078ae4d8fc0b8"
server_key_iv: "b27fa28331cc0d09951fd677a33d25e0"
server_key_version: "encryptionkey@0"
I20260927 11:18:02.669158 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260927 11:18:02.670080 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:02.670260 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:02.670351 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "b5bc97cb357f4ec588b2cf796085cda1"
format_stamp: "Formatted at 2026-09-27 11:18:02 on dist-test-slave-lsvg"
server_key: "b5df56a9a3799870fe6078ae4d8fc0b8"
server_key_iv: "b27fa28331cc0d09951fd677a33d25e0"
server_key_version: "encryptionkey@0"
I20260927 11:18:02.670428 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:02.683882 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:02.686213 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:02.686369 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:02.686621 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:02.686976 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:02.687023 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:02.687048 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:02.687060 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:02.693164 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44587
I20260927 11:18:02.693233 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44587 every 8 connection(s)
I20260927 11:18:02.693569 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:18:02.695782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 1024
I20260927 11:18:02.695889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:02.696049 392 external_mini_cluster.cc:1428] Setting key 9ff57c838953b25ad44a528467a5ea92
Sep 27 11:18:02 dist-test-slave-lsvg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:02.704129 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:02.694323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57523 (local address 127.0.98.62:37343)
0927 11:18:02.694397 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:02.694399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:02.694636 (+ 237us) server_negotiation.cc:407] Connection header received
0927 11:18:02.695181 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:02.695182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:02.695193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:02.695257 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:02.696405 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:02.697156 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:02.698191 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:02.698412 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:02.698491 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:02.700964 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:02.700965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:02.700966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:02.700975 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:02.702747 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:02.703139 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:02.703140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:02.703140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:02.703203 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:02.703438 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:02.703438 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:02.703439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:02.703589 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:02.703669 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:02.703913 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 11:18:02.704025 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 11:18:02.704154 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:02.694300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37343 (local address 127.0.98.1:57523)
0927 11:18:02.694494 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:02.694538 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:02.695065 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:02.695333 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:02.695337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:02.695975 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:02.696313 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:02.696321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:02.697239 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:02.697240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:02.698089 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:02.698092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:02.698496 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:02.698497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:02.698627 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:02.699238 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:02.699248 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:02.700855 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:02.702838 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:02.702844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:02.702846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:02.703062 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:02.703272 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:02.703272 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:02.703273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:02.703372 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:02.703660 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:02.703663 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:02.703778 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 11:18:02.703861 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 11:18:02.703981 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 11:18:02.704677 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37343
I20260927 11:18:02.704763 1153 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:02.704957 1153 heartbeater.cc:511] Master 127.0.98.62:37343 requested a full tablet report, sending...
I20260927 11:18:02.705575 962 ts_manager.cc:194] Registered new tserver with Master: b5bc97cb357f4ec588b2cf796085cda1 (127.0.98.1:44587)
I20260927 11:18:02.706208 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:18:02.725126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37343
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:02.806533 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:02.806764 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:02.806790 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:02.807951 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:02.808000 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:02.808028 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:02.809434 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:02.809633 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:02.809819 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:02.814901 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:02.815052 1158 server_base.cc:1034] running on GCE node
W20260927 11:18:02.814934 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:02.814901 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:02.815325 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:02.815609 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:02.816808 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790507882816794 us; error 44 us; skew 500 ppm
Sep 27 11:18:02 dist-test-slave-lsvg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:02.819527 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:18:02.820075 1158 webserver.cc:460] Webserver started at http://127.0.98.2:35509/ using document root <none> and password file <none>
I20260927 11:18:02.820323 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:02.820406 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:02.820565 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:02.821753 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "a0addddb9f20454f925d7afc1e9aab3d"
format_stamp: "Formatted at 2026-09-27 11:18:02 on dist-test-slave-lsvg"
server_key: "bbc41dc3b6d40e46d346d9c5c588b6e8"
server_key_iv: "a46d56e77ce0a8ac0a4495adf539297b"
server_key_version: "encryptionkey@0"
I20260927 11:18:02.822131 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a0addddb9f20454f925d7afc1e9aab3d"
format_stamp: "Formatted at 2026-09-27 11:18:02 on dist-test-slave-lsvg"
server_key: "bbc41dc3b6d40e46d346d9c5c588b6e8"
server_key_iv: "a46d56e77ce0a8ac0a4495adf539297b"
server_key_version: "encryptionkey@0"
I20260927 11:18:02.823500 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:18:02.824318 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:02.824544 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:02.824666 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "a0addddb9f20454f925d7afc1e9aab3d"
format_stamp: "Formatted at 2026-09-27 11:18:02 on dist-test-slave-lsvg"
server_key: "bbc41dc3b6d40e46d346d9c5c588b6e8"
server_key_iv: "a46d56e77ce0a8ac0a4495adf539297b"
server_key_version: "encryptionkey@0"
I20260927 11:18:02.824744 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:02.831300 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:02.833608 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:02.833799 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:02.834043 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:02.834389 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:02.834447 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:02.834483 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:02.834512 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:02.840400 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40519
I20260927 11:18:02.840437 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40519 every 8 connection(s)
I20260927 11:18:02.840817 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:18:02 dist-test-slave-lsvg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:02.849668 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:02.841635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41503 (local address 127.0.98.62:37343)
0927 11:18:02.841705 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:02.841707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:02.841918 (+ 211us) server_negotiation.cc:407] Connection header received
0927 11:18:02.842422 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:02.842422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:02.842432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:02.842492 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:02.843390 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:02.843872 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:02.844836 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:02.845017 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:02.845131 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:02.847175 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:02.847176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:02.847176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:02.847183 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:02.848570 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:02.848904 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:02.848905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:02.848905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:02.848953 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:02.849156 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:02.849156 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:02.849157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:02.849256 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:02.849308 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:02.849531 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 11:18:02.849591 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 11:18:02.849860 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:02.841624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37343 (local address 127.0.98.2:41503)
0927 11:18:02.841788 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:02.841829 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:02.842335 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:02.842549 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:02.842552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:02.843117 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:02.843311 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:02.843316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:02.843947 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:02.843948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:02.844754 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:02.844756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:02.845135 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:02.845136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:02.845257 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:02.845726 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:02.845734 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:02.847108 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:02.848640 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:02.848644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:02.848646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:02.848842 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:02.849008 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:02.849009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:02.849009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:02.849114 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:02.849311 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:02.849314 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:02.849424 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 11:18:02.849503 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 11:18:02.849618 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260927 11:18:02.849939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 1158
I20260927 11:18:02.850028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:02.850209 392 external_mini_cluster.cc:1428] Setting key 91ee37e99cfe246cf96cf3efefa29cc2
I20260927 11:18:02.850397 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37343
I20260927 11:18:02.850510 1287 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:02.850701 1287 heartbeater.cc:511] Master 127.0.98.62:37343 requested a full tablet report, sending...
I20260927 11:18:02.851169 962 ts_manager.cc:194] Registered new tserver with Master: a0addddb9f20454f925d7afc1e9aab3d (127.0.98.2:40519)
I20260927 11:18:02.851671 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41503
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:18:02.877487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37343
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:02.957512 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:02.957720 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:02.957742 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:02.958911 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:02.958952 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:02.958978 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:02.960364 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:02.960599 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:02.960813 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:02.965701 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:02.965794 1292 server_base.cc:1034] running on GCE node
W20260927 11:18:02.965715 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:02.965790 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:02.966071 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:02.966281 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:02.967448 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790507882967430 us; error 40 us; skew 500 ppm
Sep 27 11:18:02 dist-test-slave-lsvg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:02.970135 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:18:02.970664 1292 webserver.cc:460] Webserver started at http://127.0.98.3:34475/ using document root <none> and password file <none>
I20260927 11:18:02.970898 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:02.970942 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:02.971082 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:02.972144 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "b57b37be879f4e618cdfd7665df2ae45"
format_stamp: "Formatted at 2026-09-27 11:18:02 on dist-test-slave-lsvg"
server_key: "5efe423e72fdd381239eff2f478f715c"
server_key_iv: "a989b9a4708c14c9f19b5a41dcb5343d"
server_key_version: "encryptionkey@0"
I20260927 11:18:02.972507 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b57b37be879f4e618cdfd7665df2ae45"
format_stamp: "Formatted at 2026-09-27 11:18:02 on dist-test-slave-lsvg"
server_key: "5efe423e72fdd381239eff2f478f715c"
server_key_iv: "a989b9a4708c14c9f19b5a41dcb5343d"
server_key_version: "encryptionkey@0"
I20260927 11:18:02.974073 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:02.975052 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:02.975301 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:02.975385 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "b57b37be879f4e618cdfd7665df2ae45"
format_stamp: "Formatted at 2026-09-27 11:18:02 on dist-test-slave-lsvg"
server_key: "5efe423e72fdd381239eff2f478f715c"
server_key_iv: "a989b9a4708c14c9f19b5a41dcb5343d"
server_key_version: "encryptionkey@0"
I20260927 11:18:02.975477 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:02.991966 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:02.993952 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:02.994187 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:02.994455 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:02.994809 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:02.994863 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:02.994895 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:02.994915 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:03.002276 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37297
I20260927 11:18:03.002635 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:03.002907 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37297 every 8 connection(s)
Sep 27 11:18:03 dist-test-slave-lsvg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:03.012190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 1292
I20260927 11:18:03.012271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:03.012382 392 external_mini_cluster.cc:1428] Setting key 74d4681458d7f9ab09b4d5056da55b76
I20260927 11:18:03.014026 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.004217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33001 (local address 127.0.98.62:37343)
0927 11:18:03.004273 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:03.004275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:03.004525 (+ 250us) server_negotiation.cc:407] Connection header received
0927 11:18:03.005246 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:03.005248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:03.005261 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:03.005336 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:03.006613 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.007435 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.008443 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.008655 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.008742 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.010839 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:03.010840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:03.010841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:03.010849 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:03.012528 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:03.013092 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:03.013094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:03.013094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:03.013161 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:03.013460 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:03.013461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:03.013461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:03.013568 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:03.013618 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:03.013862 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 11:18:03.013963 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:18:03.014142 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.004201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37343 (local address 127.0.98.3:33001)
0927 11:18:03.004388 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:03.004435 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:03.005114 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:03.005425 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:03.005429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:03.006247 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:03.006514 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:03.006519 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.007535 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.007536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:03.008349 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:03.008351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.008751 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.008752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:03.008891 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.009382 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:03.009392 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:03.010740 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:03.012645 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:03.012651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:03.012654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:03.013004 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:03.013226 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:03.013226 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:03.013227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:03.013385 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:03.013623 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:03.013626 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:03.013743 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 11:18:03.013822 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 11:18:03.013943 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260927 11:18:03.014595 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37343
I20260927 11:18:03.014678 1421 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:03.014817 1421 heartbeater.cc:511] Master 127.0.98.62:37343 requested a full tablet report, sending...
I20260927 11:18:03.015154 962 ts_manager.cc:194] Registered new tserver with Master: b57b37be879f4e618cdfd7665df2ae45 (127.0.98.3:37297)
I20260927 11:18:03.015539 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33001
I20260927 11:18:03.024425 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:18:03.032475 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.026119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.0.98.62:37343)
0927 11:18:03.026188 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:03.026190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:03.026271 (+ 81us) server_negotiation.cc:407] Connection header received
0927 11:18:03.026312 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:03.026313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:03.026323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:03.026404 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:03.026953 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.027521 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.028247 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.028458 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.028581 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.029889 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:03.029890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:03.029890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:03.029897 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:03.031294 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:03.031658 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:03.031659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:03.031659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:03.031710 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:03.031890 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:03.031891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:03.031891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:03.032036 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:03.032113 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:03.032337 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 11:18:03.032396 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 11:18:03.035066 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:03.036684 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:03.043418 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.039214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.0.98.1:44587)
0927 11:18:03.039543 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:03.039548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:03.039567 (+ 19us) server_negotiation.cc:407] Connection header received
0927 11:18:03.039613 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:03.039615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:03.039630 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:03.039708 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:03.040062 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.040794 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.042443 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.042942 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.043031 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.043096 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:03.043271 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 11:18:03.043325 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":75,"threads_started":1}
I20260927 11:18:03.043447 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.039311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40519 (local address 127.0.0.1:41184)
0927 11:18:03.039461 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:03.039472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:03.039526 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:03.039859 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:03.039860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:03.039861 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:03.039890 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:03.039891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.041206 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.041207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:03.042335 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:03.042337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.043191 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.043192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:03.043249 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.043253 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:03.043313 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 11:18:03.043355 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":43,"thread_start_us":90,"threads_started":1}
I20260927 11:18:03.043450 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.039547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.0.98.2:40519)
0927 11:18:03.039716 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:03.039720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:03.039729 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:03.039765 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:03.039767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:03.039780 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:03.039865 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:03.039960 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.041104 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.042441 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.043129 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.043193 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.043249 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:03.043327 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 11:18:03.043368 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260927 11:18:03.043646 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.039095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44587 (local address 127.0.0.1:33164)
0927 11:18:03.039397 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:03.039407 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:03.039449 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:03.039791 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:03.039791 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:03.039793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:03.039980 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:03.039982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.040927 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.040927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:03.042335 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:03.042337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.043041 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.043042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:03.043132 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.043146 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 11:18:03.043214 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 11:18:03.043583 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":327,"thread_start_us":84,"threads_started":1}
I20260927 11:18:03.043368 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.039080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37297 (local address 127.0.0.1:42994)
0927 11:18:03.039278 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:03.039322 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:03.039391 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:03.039763 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:03.039766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:03.039769 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:03.039872 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:03.039875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.041250 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.041251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:03.042173 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:03.042175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.043100 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.043102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:03.043172 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.043177 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:18:03.043208 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 11:18:03.043280 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":24,"thread_start_us":80,"threads_started":1}
I20260927 11:18:03.044143 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.039161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.0.98.3:37297)
0927 11:18:03.039580 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:03.039584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:03.039592 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:03.039629 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:03.039631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:03.039645 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:03.039747 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:03.039969 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.041127 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.042257 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.042999 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.043411 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.043967 (+ 556us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:03.044000 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 11:18:03.044047 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":79,"threads_started":1}
I20260927 11:18:03.044267 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 9b38c9906e7048a198a2af5465ce2f4a (DEFAULT_TABLE table=test-table [id=48214641b7c149d7afdf2815b9aab79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:03.044395 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 9b38c9906e7048a198a2af5465ce2f4a (DEFAULT_TABLE table=test-table [id=48214641b7c149d7afdf2815b9aab79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:03.044667 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b38c9906e7048a198a2af5465ce2f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:03.044734 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b38c9906e7048a198a2af5465ce2f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:03.044737 1340 tablet_service.cc:1467] Processing CreateTablet for tablet 9b38c9906e7048a198a2af5465ce2f4a (DEFAULT_TABLE table=test-table [id=48214641b7c149d7afdf2815b9aab79a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:03.045086 1340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b38c9906e7048a198a2af5465ce2f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:03.048007 1438 tablet_bootstrap.cc:492] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d: Bootstrap starting.
I20260927 11:18:03.048300 1439 tablet_bootstrap.cc:492] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1: Bootstrap starting.
I20260927 11:18:03.048450 1440 tablet_bootstrap.cc:492] T 9b38c9906e7048a198a2af5465ce2f4a P b57b37be879f4e618cdfd7665df2ae45: Bootstrap starting.
I20260927 11:18:03.048914 1439 tablet_bootstrap.cc:654] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:03.048914 1438 tablet_bootstrap.cc:654] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:03.049031 1440 tablet_bootstrap.cc:654] T 9b38c9906e7048a198a2af5465ce2f4a P b57b37be879f4e618cdfd7665df2ae45: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:03.049539 1440 log.cc:824] T 9b38c9906e7048a198a2af5465ce2f4a P b57b37be879f4e618cdfd7665df2ae45: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:03.049571 1439 log.cc:824] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:03.049571 1438 log.cc:824] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:03.050263 1438 tablet_bootstrap.cc:492] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d: No bootstrap required, opened a new log
I20260927 11:18:03.050263 1439 tablet_bootstrap.cc:492] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1: No bootstrap required, opened a new log
I20260927 11:18:03.050320 1438 ts_tablet_manager.cc:1397] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 11:18:03.050344 1439 ts_tablet_manager.cc:1397] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:18:03.051882 1439 raft_consensus.cc:348] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b37be879f4e618cdfd7665df2ae45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37297 } } peers { permanent_uuid: "a0addddb9f20454f925d7afc1e9aab3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40519 } } peers { permanent_uuid: "b5bc97cb357f4ec588b2cf796085cda1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44587 } }
I20260927 11:18:03.052053 1439 raft_consensus.cc:374] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:03.052099 1439 raft_consensus.cc:729] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5bc97cb357f4ec588b2cf796085cda1, State: Initialized, Role: FOLLOWER
I20260927 11:18:03.052227 1439 consensus_queue.cc:260] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b37be879f4e618cdfd7665df2ae45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37297 } } peers { permanent_uuid: "a0addddb9f20454f925d7afc1e9aab3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40519 } } peers { permanent_uuid: "b5bc97cb357f4ec588b2cf796085cda1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44587 } }
I20260927 11:18:03.052363 1438 raft_consensus.cc:348] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b37be879f4e618cdfd7665df2ae45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37297 } } peers { permanent_uuid: "a0addddb9f20454f925d7afc1e9aab3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40519 } } peers { permanent_uuid: "b5bc97cb357f4ec588b2cf796085cda1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44587 } }
I20260927 11:18:03.052552 1438 raft_consensus.cc:374] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:03.052567 1439 ts_tablet_manager.cc:1428] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:18:03.052585 1153 heartbeater.cc:503] Master 127.0.98.62:37343 was elected leader, sending a full tablet report...
I20260927 11:18:03.052598 1438 raft_consensus.cc:729] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0addddb9f20454f925d7afc1e9aab3d, State: Initialized, Role: FOLLOWER
I20260927 11:18:03.052708 1438 consensus_queue.cc:260] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b37be879f4e618cdfd7665df2ae45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37297 } } peers { permanent_uuid: "a0addddb9f20454f925d7afc1e9aab3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40519 } } peers { permanent_uuid: "b5bc97cb357f4ec588b2cf796085cda1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44587 } }
I20260927 11:18:03.052982 1438 ts_tablet_manager.cc:1428] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 11:18:03.053035 1440 tablet_bootstrap.cc:492] T 9b38c9906e7048a198a2af5465ce2f4a P b57b37be879f4e618cdfd7665df2ae45: No bootstrap required, opened a new log
I20260927 11:18:03.053128 1287 heartbeater.cc:503] Master 127.0.98.62:37343 was elected leader, sending a full tablet report...
I20260927 11:18:03.053149 1440 ts_tablet_manager.cc:1397] T 9b38c9906e7048a198a2af5465ce2f4a P b57b37be879f4e618cdfd7665df2ae45: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260927 11:18:03.054778 1440 raft_consensus.cc:348] T 9b38c9906e7048a198a2af5465ce2f4a P b57b37be879f4e618cdfd7665df2ae45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b37be879f4e618cdfd7665df2ae45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37297 } } peers { permanent_uuid: "a0addddb9f20454f925d7afc1e9aab3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40519 } } peers { permanent_uuid: "b5bc97cb357f4ec588b2cf796085cda1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44587 } }
I20260927 11:18:03.054992 1440 raft_consensus.cc:374] T 9b38c9906e7048a198a2af5465ce2f4a P b57b37be879f4e618cdfd7665df2ae45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:03.055039 1440 raft_consensus.cc:729] T 9b38c9906e7048a198a2af5465ce2f4a P b57b37be879f4e618cdfd7665df2ae45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57b37be879f4e618cdfd7665df2ae45, State: Initialized, Role: FOLLOWER
I20260927 11:18:03.055155 1440 consensus_queue.cc:260] T 9b38c9906e7048a198a2af5465ce2f4a P b57b37be879f4e618cdfd7665df2ae45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b37be879f4e618cdfd7665df2ae45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37297 } } peers { permanent_uuid: "a0addddb9f20454f925d7afc1e9aab3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40519 } } peers { permanent_uuid: "b5bc97cb357f4ec588b2cf796085cda1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44587 } }
I20260927 11:18:03.055437 1440 ts_tablet_manager.cc:1428] T 9b38c9906e7048a198a2af5465ce2f4a P b57b37be879f4e618cdfd7665df2ae45: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 11:18:03.055455 1421 heartbeater.cc:503] Master 127.0.98.62:37343 was elected leader, sending a full tablet report...
I20260927 11:18:03.070533 1444 raft_consensus.cc:482] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:03.070652 1444 raft_consensus.cc:504] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b37be879f4e618cdfd7665df2ae45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37297 } } peers { permanent_uuid: "a0addddb9f20454f925d7afc1e9aab3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40519 } } peers { permanent_uuid: "b5bc97cb357f4ec588b2cf796085cda1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44587 } }
I20260927 11:18:03.071050 1444 leader_election.cc:288] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b57b37be879f4e618cdfd7665df2ae45 (127.0.98.3:37297), a0addddb9f20454f925d7afc1e9aab3d (127.0.98.2:40519)
I20260927 11:18:03.074398 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.071354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40519 (local address 127.0.98.1:33831)
0927 11:18:03.071541 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:03.071552 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:03.071610 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:03.071747 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:03.071748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:03.071749 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:03.071810 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:03.071811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.072563 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.072564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:03.073518 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:03.073521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.074138 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.074140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:03.074213 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.074217 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:03.074259 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 11:18:03.074290 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":121,"threads_started":1}
I20260927 11:18:03.074414 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.071479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33831 (local address 127.0.98.2:40519)
0927 11:18:03.071527 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:03.071531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:03.071618 (+ 87us) server_negotiation.cc:407] Connection header received
0927 11:18:03.071664 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:03.071665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:03.071677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:03.071730 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:03.071915 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.072466 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.073608 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.074051 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.074137 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.074150 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:03.074292 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 11:18:03.074343 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 11:18:03.074482 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.071314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37169 (local address 127.0.98.3:37297)
0927 11:18:03.071370 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:03.071372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:03.071407 (+ 35us) server_negotiation.cc:407] Connection header received
0927 11:18:03.071456 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:03.071456 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:03.071490 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:03.071547 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:03.071748 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.072289 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.073508 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.074059 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.074149 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.074158 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:03.074348 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 11:18:03.074411 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 11:18:03.074604 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.071235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37297 (local address 127.0.98.1:37169)
0927 11:18:03.071294 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:03.071306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:03.071351 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:03.071573 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:03.071573 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:03.071575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:03.071666 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:03.071668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.072401 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.072402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:03.073411 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:03.073414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.074162 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.074163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:03.074264 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.074269 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:18:03.074334 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 11:18:03.074384 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":4}
I20260927 11:18:03.075078 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b38c9906e7048a198a2af5465ce2f4a" candidate_uuid: "b5bc97cb357f4ec588b2cf796085cda1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57b37be879f4e618cdfd7665df2ae45" is_pre_election: true
I20260927 11:18:03.075109 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b38c9906e7048a198a2af5465ce2f4a" candidate_uuid: "b5bc97cb357f4ec588b2cf796085cda1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0addddb9f20454f925d7afc1e9aab3d" is_pre_election: true
I20260927 11:18:03.075270 1375 raft_consensus.cc:2454] T 9b38c9906e7048a198a2af5465ce2f4a P b57b37be879f4e618cdfd7665df2ae45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5bc97cb357f4ec588b2cf796085cda1 in term 0.
I20260927 11:18:03.075284 1241 raft_consensus.cc:2454] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5bc97cb357f4ec588b2cf796085cda1 in term 0.
I20260927 11:18:03.075507 1042 leader_election.cc:302] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b57b37be879f4e618cdfd7665df2ae45, b5bc97cb357f4ec588b2cf796085cda1; no voters:
I20260927 11:18:03.075667 1444 raft_consensus.cc:2781] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:03.075747 1444 raft_consensus.cc:482] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:03.075783 1444 raft_consensus.cc:3037] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:03.076550 1444 raft_consensus.cc:504] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b37be879f4e618cdfd7665df2ae45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37297 } } peers { permanent_uuid: "a0addddb9f20454f925d7afc1e9aab3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40519 } } peers { permanent_uuid: "b5bc97cb357f4ec588b2cf796085cda1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44587 } }
I20260927 11:18:03.076704 1444 leader_election.cc:288] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [CANDIDATE]: Term 1 election: Requested vote from peers b57b37be879f4e618cdfd7665df2ae45 (127.0.98.3:37297), a0addddb9f20454f925d7afc1e9aab3d (127.0.98.2:40519)
I20260927 11:18:03.076921 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b38c9906e7048a198a2af5465ce2f4a" candidate_uuid: "b5bc97cb357f4ec588b2cf796085cda1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0addddb9f20454f925d7afc1e9aab3d"
I20260927 11:18:03.076915 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b38c9906e7048a198a2af5465ce2f4a" candidate_uuid: "b5bc97cb357f4ec588b2cf796085cda1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b57b37be879f4e618cdfd7665df2ae45"
I20260927 11:18:03.077024 1241 raft_consensus.cc:3037] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:03.077020 1375 raft_consensus.cc:3037] T 9b38c9906e7048a198a2af5465ce2f4a P b57b37be879f4e618cdfd7665df2ae45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:03.077745 1375 raft_consensus.cc:2454] T 9b38c9906e7048a198a2af5465ce2f4a P b57b37be879f4e618cdfd7665df2ae45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5bc97cb357f4ec588b2cf796085cda1 in term 1.
I20260927 11:18:03.077864 1241 raft_consensus.cc:2454] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5bc97cb357f4ec588b2cf796085cda1 in term 1.
I20260927 11:18:03.077930 1042 leader_election.cc:302] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b57b37be879f4e618cdfd7665df2ae45, b5bc97cb357f4ec588b2cf796085cda1; no voters:
I20260927 11:18:03.078009 1444 raft_consensus.cc:2781] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:03.078161 1444 raft_consensus.cc:686] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [term 1 LEADER]: Becoming Leader. State: Replica: b5bc97cb357f4ec588b2cf796085cda1, State: Running, Role: LEADER
I20260927 11:18:03.078258 1444 consensus_queue.cc:237] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b37be879f4e618cdfd7665df2ae45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37297 } } peers { permanent_uuid: "a0addddb9f20454f925d7afc1e9aab3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40519 } } peers { permanent_uuid: "b5bc97cb357f4ec588b2cf796085cda1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44587 } }
I20260927 11:18:03.079166 962 catalog_manager.cc:5495] T 9b38c9906e7048a198a2af5465ce2f4a P b5bc97cb357f4ec588b2cf796085cda1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5bc97cb357f4ec588b2cf796085cda1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b5bc97cb357f4ec588b2cf796085cda1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57b37be879f4e618cdfd7665df2ae45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0addddb9f20454f925d7afc1e9aab3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5bc97cb357f4ec588b2cf796085cda1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44587 } health_report { overall_health: HEALTHY } } }
Sep 27 11:18:03 dist-test-slave-lsvg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
W20260927 11:18:03.091431 1288 tablet.cc:2369] T 9b38c9906e7048a198a2af5465ce2f4a P a0addddb9f20454f925d7afc1e9aab3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:03.093662 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.085832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.0.98.1:44587)
0927 11:18:03.085913 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:03.085915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:03.086017 (+ 102us) server_negotiation.cc:407] Connection header received
0927 11:18:03.086061 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:03.086061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:03.086074 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:03.086134 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:03.086688 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.087280 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.088046 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.088326 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.088401 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.090810 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:03.090820 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:03.090820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:03.090827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:03.092467 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:03.092838 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:03.092841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:03.092842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:03.092901 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:03.093168 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:03.093169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:03.093170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:03.093310 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:03.093420 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:03.093484 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 11:18:03.093549 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 11:18:03 dist-test-slave-lsvg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 11:18:03 dist-test-slave-lsvg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:18:03.116988 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.109824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.0.98.1:44587)
0927 11:18:03.109897 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:03.109899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:03.109976 (+ 77us) server_negotiation.cc:407] Connection header received
0927 11:18:03.110005 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:03.110006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:03.110015 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:03.110067 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:03.110631 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.111134 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.111897 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.112098 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.112148 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.114613 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:03.114614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:03.114615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:03.114622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:03.116069 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:03.116357 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:03.116358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:03.116358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:03.116408 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:03.116557 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:03.116558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:03.116558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:03.116639 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:03.116708 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:03.116832 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 11:18:03.116921 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 11:18:03 dist-test-slave-lsvg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 11:18:03 dist-test-slave-lsvg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:18:03.139890 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:03.132452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.0.98.1:44587)
0927 11:18:03.132510 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:03.132512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:03.132679 (+ 167us) server_negotiation.cc:407] Connection header received
0927 11:18:03.132765 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:03.132767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:03.132783 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:03.132849 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:03.133545 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.134025 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.134785 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:03.134987 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:03.135075 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:03.137567 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:03.137569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:03.137570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:03.137585 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:03.138748 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:03.139118 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:03.139120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:03.139120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:03.139180 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:03.139377 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:03.139378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:03.139378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:03.139486 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:03.139600 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:03.139699 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 11:18:03.139783 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 11:18:03.140413 1067 tablet_service.cc:2908] Rejecting scan request for tablet 9b38c9906e7048a198a2af5465ce2f4a: Uninitialized: clean time has not yet been initialized
I20260927 11:18:03.141458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 1024
I20260927 11:18:03.146298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 1158
I20260927 11:18:03.150708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 1292
I20260927 11:18:03.155043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 930
2026-09-27T11:18:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1958 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:03 dist-test-slave-lsvg krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:03 dist-test-slave-lsvg krb5kdc[1462](info): set up 2 sockets
Sep 27 11:18:03 dist-test-slave-lsvg krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260927 11:18:04.192579 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:04 dist-test-slave-lsvg krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:18:04.315366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40459
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:04.399358 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:04.399586 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:04.399610 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:04.400803 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:04.400851 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 11:18:04.400866 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:04.400880 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:04.400894 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:04.400908 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:04.402412 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40459
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:04.402609 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:04.402789 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:04.407727 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:04.407707 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:04.407738 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:04.407790 1475 server_base.cc:1034] running on GCE node
I20260927 11:18:04.408115 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:04.408349 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:04.409485 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790507884409473 us; error 31 us; skew 500 ppm
Sep 27 11:18:04 dist-test-slave-lsvg krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:04.411984 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:18:04.412397 1475 webserver.cc:460] Webserver started at http://127.0.98.62:33267/ using document root <none> and password file <none>
I20260927 11:18:04.412614 1475 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:04.412680 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:04.412838 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:04.413950 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "1bc08646ced84351ae8ced0376aa8795"
format_stamp: "Formatted at 2026-09-27 11:18:04 on dist-test-slave-lsvg"
server_key: "38a3b4a79fb245b708de618446513140"
server_key_iv: "c8f8d69900321d0a0f0c531c505374e9"
server_key_version: "encryptionkey@0"
I20260927 11:18:04.414307 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "1bc08646ced84351ae8ced0376aa8795"
format_stamp: "Formatted at 2026-09-27 11:18:04 on dist-test-slave-lsvg"
server_key: "38a3b4a79fb245b708de618446513140"
server_key_iv: "c8f8d69900321d0a0f0c531c505374e9"
server_key_version: "encryptionkey@0"
I20260927 11:18:04.415638 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:18:04.416429 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:04.416612 1475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:04.416712 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "1bc08646ced84351ae8ced0376aa8795"
format_stamp: "Formatted at 2026-09-27 11:18:04 on dist-test-slave-lsvg"
server_key: "38a3b4a79fb245b708de618446513140"
server_key_iv: "c8f8d69900321d0a0f0c531c505374e9"
server_key_version: "encryptionkey@0"
I20260927 11:18:04.416797 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:04.426647 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:04.428910 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:04.429126 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:04.433560 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40459
I20260927 11:18:04.433665 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40459 every 8 connection(s)
I20260927 11:18:04.433948 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:04.434630 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:04.436872 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795: Bootstrap starting.
I20260927 11:18:04.437512 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:04.437759 1553 log.cc:824] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:04.438354 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795: No bootstrap required, opened a new log
I20260927 11:18:04.440346 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc08646ced84351ae8ced0376aa8795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } }
I20260927 11:18:04.440564 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:04.440580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 1475
I20260927 11:18:04.440606 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc08646ced84351ae8ced0376aa8795, State: Initialized, Role: FOLLOWER
I20260927 11:18:04.440692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:04.440757 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc08646ced84351ae8ced0376aa8795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } }
I20260927 11:18:04.440842 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:04.440881 392 external_mini_cluster.cc:1428] Setting key 12899e8db5986f9d22f44bae6c7b1b6a
I20260927 11:18:04.440886 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:04.440936 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:04.441700 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc08646ced84351ae8ced0376aa8795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } }
I20260927 11:18:04.441797 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bc08646ced84351ae8ced0376aa8795; no voters:
I20260927 11:18:04.442008 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:04.442040 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:04.442155 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [term 1 LEADER]: Becoming Leader. State: Replica: 1bc08646ced84351ae8ced0376aa8795, State: Running, Role: LEADER
I20260927 11:18:04.442238 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc08646ced84351ae8ced0376aa8795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } }
I20260927 11:18:04.442276 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:04.442538 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc08646ced84351ae8ced0376aa8795. Latest consensus state: current_term: 1 leader_uuid: "1bc08646ced84351ae8ced0376aa8795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc08646ced84351ae8ced0376aa8795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } }
I20260927 11:18:04.442623 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:04.442550 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bc08646ced84351ae8ced0376aa8795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc08646ced84351ae8ced0376aa8795" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40459 } } }
I20260927 11:18:04.442909 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:04.443152 1566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:04.443686 1566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:04.445339 1566 catalog_manager.cc:1347] Generated new cluster ID: 9b5f918c886e4fd982ab396fce153478
I20260927 11:18:04.445395 1566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:18:04 dist-test-slave-lsvg krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:04.453953 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:04.441426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.0.98.62:40459)
0927 11:18:04.441605 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:04.441614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:04.441649 (+ 35us) server_negotiation.cc:407] Connection header received
0927 11:18:04.441740 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:04.441745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:04.441822 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:04.441960 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:04.442793 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:04.444518 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:04.445571 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:04.446182 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:04.446284 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:04.449181 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:04.449185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:04.449186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:04.449197 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:04.451818 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:04.452234 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:04.452237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:04.452238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:04.452363 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:04.452585 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:04.452586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:04.452586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:04.452889 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:04.453510 (+ 621us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:04.453587 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 11:18:04.453730 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
I20260927 11:18:04.460927 1566 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:04.461474 1566 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:04.474853 1566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795: Generated new TSK 0
I20260927 11:18:04.475128 1566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260927 11:18:05.027279 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.543s user 0.000s sys 0.003s
I20260927 11:18:05.027633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40459
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:05.108501 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:05.108718 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:05.108743 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:05.109895 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:05.109942 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:05.109965 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:18:05.109984 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:05.111382 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:05.111573 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:05.111819 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:18:05.116830 1579 server_base.cc:1034] running on GCE node
W20260927 11:18:05.116781 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:05.116782 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:05.116796 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:05.117213 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:05.117445 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:05.118595 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790507885118594 us; error 57 us; skew 500 ppm
Sep 27 11:18:05 dist-test-slave-lsvg krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:05.121518 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:18:05.121955 1579 webserver.cc:460] Webserver started at http://127.0.98.1:33511/ using document root <none> and password file <none>
I20260927 11:18:05.122174 1579 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:05.122237 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:05.122364 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:05.123389 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "952d92ec4b9f4188b3d6c690bb5feab2"
format_stamp: "Formatted at 2026-09-27 11:18:05 on dist-test-slave-lsvg"
server_key: "1ce83a4b3c3bc0689117eeafc3e1366a"
server_key_iv: "0651c8acf75ccf0f8edac9d3af76b826"
server_key_version: "encryptionkey@0"
I20260927 11:18:05.123749 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "952d92ec4b9f4188b3d6c690bb5feab2"
format_stamp: "Formatted at 2026-09-27 11:18:05 on dist-test-slave-lsvg"
server_key: "1ce83a4b3c3bc0689117eeafc3e1366a"
server_key_iv: "0651c8acf75ccf0f8edac9d3af76b826"
server_key_version: "encryptionkey@0"
I20260927 11:18:05.125113 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:05.125872 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:05.126029 1579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:05.126121 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "952d92ec4b9f4188b3d6c690bb5feab2"
format_stamp: "Formatted at 2026-09-27 11:18:05 on dist-test-slave-lsvg"
server_key: "1ce83a4b3c3bc0689117eeafc3e1366a"
server_key_iv: "0651c8acf75ccf0f8edac9d3af76b826"
server_key_version: "encryptionkey@0"
I20260927 11:18:05.126199 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:05.155906 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:05.158285 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:05.158442 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:05.159030 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:05.159093 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:05.159135 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:05.159166 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 11:18:05 dist-test-slave-lsvg krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:05.166290 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37831
I20260927 11:18:05.166400 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37831 every 8 connection(s)
I20260927 11:18:05.166666 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:18:05.169683 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.159518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54147 (local address 127.0.98.62:40459)
0927 11:18:05.159642 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.159644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.159837 (+ 193us) server_negotiation.cc:407] Connection header received
0927 11:18:05.160378 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.160380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.160393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.160476 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:05.161470 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.162332 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.164016 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.164352 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.164418 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.166775 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:05.166777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:05.166777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:05.166786 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:05.168287 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.168678 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.168679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.168679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.168743 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.168992 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.168993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.168994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.169157 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:05.169243 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.169507 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.169588 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260927 11:18:05.169863 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.159497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:54147)
0927 11:18:05.159678 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.159748 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.160278 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.160594 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.160599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.161012 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:05.161361 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.161370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.162449 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.162450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.163919 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.163922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.164445 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.164446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.164598 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.165217 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:05.165232 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:05.166665 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:05.168385 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.168391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.168393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.168605 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.168812 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.168812 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.168813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.168921 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.169236 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:05.169242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:05.169374 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.169474 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.169625 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260927 11:18:05.170346 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40459
I20260927 11:18:05.170423 1711 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:05.170604 1711 heartbeater.cc:511] Master 127.0.98.62:40459 requested a full tablet report, sending...
I20260927 11:18:05.171180 1507 ts_manager.cc:194] Registered new tserver with Master: 952d92ec4b9f4188b3d6c690bb5feab2 (127.0.98.1:37831)
I20260927 11:18:05.171804 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54147
I20260927 11:18:05.172566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 1579
I20260927 11:18:05.172677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:05.172847 392 external_mini_cluster.cc:1428] Setting key 36c210611611ea42bb3dc485e9cb1c40
I20260927 11:18:05.177821 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.171345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.1:44449)
0927 11:18:05.171478 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.171488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.171538 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.172146 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.172147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.172316 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:05.172362 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.172363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.173170 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.173171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.173974 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.173977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.174296 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.174297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.174365 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.174854 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:05.174860 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:05.175127 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:05.176835 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.176836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.176836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.177054 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.177266 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.177267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.177267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.177317 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.177579 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:05.177579 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:05.177611 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.177657 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.177751 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":86,"threads_started":1}
I20260927 11:18:05.177881 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.171991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44449 (local address 127.0.98.62:40459)
0927 11:18:05.172037 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.172039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.172047 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:05.172064 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.172065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.172074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.172128 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:05.172432 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.173094 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.174048 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.174229 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.174282 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.175203 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:05.175204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:05.175204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:05.175212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:05.176750 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.177144 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.177145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.177146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.177207 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.177377 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.177378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.177379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.177513 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:05.177575 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.177683 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.177775 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:18:05.200721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40459
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:05.280503 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:05.280732 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:05.280768 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:05.281929 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:05.281993 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:05.282024 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:18:05.282063 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:05.283546 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:05.283799 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:05.284013 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:05.288932 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:05.289039 1720 server_base.cc:1034] running on GCE node
W20260927 11:18:05.288964 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:05.288950 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:05.289390 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:05.289589 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:05.290721 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790507885290703 us; error 38 us; skew 500 ppm
Sep 27 11:18:05 dist-test-slave-lsvg krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:05.297180 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:18:05.297617 1720 webserver.cc:460] Webserver started at http://127.0.98.2:35195/ using document root <none> and password file <none>
I20260927 11:18:05.297855 1720 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:05.297919 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:05.298060 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:05.299101 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "bae9c409c0d94ab7bc734bc4fbdaa949"
format_stamp: "Formatted at 2026-09-27 11:18:05 on dist-test-slave-lsvg"
server_key: "bdb54b49288c41d2fa2a5586b457f1fe"
server_key_iv: "673ee4d654f7725433c954f581494d8b"
server_key_version: "encryptionkey@0"
I20260927 11:18:05.299466 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bae9c409c0d94ab7bc734bc4fbdaa949"
format_stamp: "Formatted at 2026-09-27 11:18:05 on dist-test-slave-lsvg"
server_key: "bdb54b49288c41d2fa2a5586b457f1fe"
server_key_iv: "673ee4d654f7725433c954f581494d8b"
server_key_version: "encryptionkey@0"
I20260927 11:18:05.300757 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:18:05.301510 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:05.301673 1720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:05.301759 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "bae9c409c0d94ab7bc734bc4fbdaa949"
format_stamp: "Formatted at 2026-09-27 11:18:05 on dist-test-slave-lsvg"
server_key: "bdb54b49288c41d2fa2a5586b457f1fe"
server_key_iv: "673ee4d654f7725433c954f581494d8b"
server_key_version: "encryptionkey@0"
I20260927 11:18:05.301838 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:05.314925 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:05.317294 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:05.317456 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:05.318018 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:05.318074 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:05.318116 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:05.318137 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 11:18:05 dist-test-slave-lsvg krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:05.325304 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44493
I20260927 11:18:05.325362 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44493 every 8 connection(s)
I20260927 11:18:05.325726 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:18:05.328048 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.318397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52667 (local address 127.0.98.62:40459)
0927 11:18:05.318462 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.318465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.318649 (+ 184us) server_negotiation.cc:407] Connection header received
0927 11:18:05.319172 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.319173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.319187 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.319273 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:05.320031 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.320847 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.322569 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.322837 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.322905 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.325452 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:05.325453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:05.325454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:05.325462 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:05.326965 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.327334 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.327335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.327335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.327385 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.327580 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.327580 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.327580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.327679 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:05.327729 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.327930 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.327986 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:18:05.328289 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.318374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:52667)
0927 11:18:05.318513 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.318554 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.319071 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.319373 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.319376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.319687 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:05.319932 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.319937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.320946 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.320947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.322481 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.322484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.322911 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.322912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.323072 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.323601 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:05.323613 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:05.325363 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:05.327050 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.327056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.327058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.327271 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.327441 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.327442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.327442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.327531 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.327731 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:05.327734 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:05.327841 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.327918 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.328030 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260927 11:18:05.328728 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40459
I20260927 11:18:05.328804 1851 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:05.328986 1851 heartbeater.cc:511] Master 127.0.98.62:40459 requested a full tablet report, sending...
I20260927 11:18:05.329393 1507 ts_manager.cc:194] Registered new tserver with Master: bae9c409c0d94ab7bc734bc4fbdaa949 (127.0.98.2:44493)
I20260927 11:18:05.329902 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52667
I20260927 11:18:05.334642 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.329868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56105 (local address 127.0.98.62:40459)
0927 11:18:05.329914 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.329916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.330063 (+ 147us) server_negotiation.cc:407] Connection header received
0927 11:18:05.330130 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.330130 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.330139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.330175 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:05.330503 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.331021 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.331653 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.331831 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.331876 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.332632 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:05.332632 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:05.332633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:05.332638 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:05.333789 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.334077 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.334077 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.334078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.334130 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.334265 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.334266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.334266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.334382 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:05.334432 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.334536 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.334586 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:18:05.334654 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.329821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.2:56105)
0927 11:18:05.330007 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.330017 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.330077 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.330188 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.330189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.330383 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:05.330436 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.330438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.331099 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.331100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.331584 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.331586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.331907 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.331908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.331966 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.332304 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:05.332308 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:05.332563 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:05.333855 (+ 1292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.333856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.333856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.334009 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.334169 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.334169 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.334170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.334221 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.334436 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:05.334437 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:05.334468 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.334519 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.334587 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":119,"threads_started":1}
I20260927 11:18:05.335341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 1720
I20260927 11:18:05.335445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:05.335655 392 external_mini_cluster.cc:1428] Setting key 979f616302a66bf8d0007fac9e7ddbd4
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:18:05.360047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40459
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:05.455351 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:05.455575 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:05.455612 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:05.456774 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:05.456837 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:05.456875 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:18:05.456924 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:05.458431 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:05.458688 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:05.458902 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:05.463735 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:05.463819 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:05.463820 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:05.463901 1860 server_base.cc:1034] running on GCE node
I20260927 11:18:05.464157 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:05.464345 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:05.465476 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790507885465457 us; error 32 us; skew 500 ppm
Sep 27 11:18:05 dist-test-slave-lsvg krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:05.467926 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:18:05.468287 1860 webserver.cc:460] Webserver started at http://127.0.98.3:46109/ using document root <none> and password file <none>
I20260927 11:18:05.468456 1860 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:05.468495 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:05.468581 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:05.469717 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "ee68b3a253ba4f4bbc98538e18f655a2"
format_stamp: "Formatted at 2026-09-27 11:18:05 on dist-test-slave-lsvg"
server_key: "ee104a9fe432e57704cf1a5b15afe12e"
server_key_iv: "f74f7e0d45e9bff3cdea0c763ef2cafa"
server_key_version: "encryptionkey@0"
I20260927 11:18:05.470026 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ee68b3a253ba4f4bbc98538e18f655a2"
format_stamp: "Formatted at 2026-09-27 11:18:05 on dist-test-slave-lsvg"
server_key: "ee104a9fe432e57704cf1a5b15afe12e"
server_key_iv: "f74f7e0d45e9bff3cdea0c763ef2cafa"
server_key_version: "encryptionkey@0"
I20260927 11:18:05.471391 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:18:05.472080 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:05.472226 1860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:05.472296 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "ee68b3a253ba4f4bbc98538e18f655a2"
format_stamp: "Formatted at 2026-09-27 11:18:05 on dist-test-slave-lsvg"
server_key: "ee104a9fe432e57704cf1a5b15afe12e"
server_key_iv: "f74f7e0d45e9bff3cdea0c763ef2cafa"
server_key_version: "encryptionkey@0"
I20260927 11:18:05.472347 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:05.491608 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:05.493991 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:05.494099 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:05.494604 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:05.494645 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:05.494668 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:05.494680 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:05.501456 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38629
I20260927 11:18:05.501519 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38629 every 8 connection(s)
Sep 27 11:18:05 dist-test-slave-lsvg krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:05.501868 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:05.504575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 1860
I20260927 11:18:05.504686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:05.504869 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.495083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57711 (local address 127.0.98.62:40459)
0927 11:18:05.495133 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.495135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.495356 (+ 221us) server_negotiation.cc:407] Connection header received
0927 11:18:05.495889 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.495891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.495903 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.495979 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:05.496987 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.497830 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.499389 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.499711 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.499803 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.502433 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:05.502434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:05.502434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:05.502441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:05.503609 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.503950 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.503951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.503951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.503998 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.504169 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.504170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.504170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.504307 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:05.504364 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.504722 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.504800 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:18:05.504885 392 external_mini_cluster.cc:1428] Setting key c43a60b5ce18cf5d2ee530713f85cb04
I20260927 11:18:05.505085 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.495082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:57711)
0927 11:18:05.495227 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.495261 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.495785 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.496099 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.496105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.496555 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:05.496872 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.496879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.497926 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.497927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.499307 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.499309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.499810 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.499811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.499922 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.500465 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:05.500481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:05.502349 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:05.503675 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.503680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.503681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.503888 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.504038 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.504039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.504039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.504120 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.504362 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:05.504367 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:05.504543 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.504669 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.504844 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260927 11:18:05.505554 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40459
I20260927 11:18:05.505640 1991 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:05.505821 1991 heartbeater.cc:511] Master 127.0.98.62:40459 requested a full tablet report, sending...
I20260927 11:18:05.506232 1507 ts_manager.cc:194] Registered new tserver with Master: ee68b3a253ba4f4bbc98538e18f655a2 (127.0.98.3:38629)
I20260927 11:18:05.506633 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57711
I20260927 11:18:05.511447 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.506675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54869 (local address 127.0.98.62:40459)
0927 11:18:05.506716 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.506718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.506842 (+ 124us) server_negotiation.cc:407] Connection header received
0927 11:18:05.506893 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.506894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.506904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.506950 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:05.507274 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.507735 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.508357 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.508523 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.508570 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.509432 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:05.509433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:05.509433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:05.509438 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:05.510520 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.510863 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.510864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.510865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.510925 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.511095 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.511095 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.511096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.511194 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:05.511249 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.511333 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.511390 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:18:05.511461 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.506602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.98.3:54869)
0927 11:18:05.506783 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.506799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.506849 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.506953 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.506954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.507150 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:05.507204 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.507205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.507819 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.507820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.508293 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.508295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.508581 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.508582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.508625 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.509092 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:05.509096 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:05.509365 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:05.510613 (+ 1248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.510614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.510614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.510780 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.510999 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.510999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.510999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.511046 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.511239 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:05.511239 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:05.511271 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.511316 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.511390 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
I20260927 11:18:05.516650 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 11:18:05 dist-test-slave-lsvg krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 11:18:05 dist-test-slave-lsvg krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:05.541931 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.533938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.0.98.62:40459)
0927 11:18:05.534007 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.534009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.534073 (+ 64us) server_negotiation.cc:407] Connection header received
0927 11:18:05.534195 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.534197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.534208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.534266 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:05.534860 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.535367 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.536104 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.536300 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.536385 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.538660 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:05.538662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:05.538662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:05.538671 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:05.540506 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.540929 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.540930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.540931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.541018 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.541294 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.541295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.541295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.541455 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:05.541560 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.541736 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.541819 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 11:18:05.543901 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:05.545733 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:05.552443 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.548197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38629 (local address 127.0.0.1:48666)
0927 11:18:05.548357 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.548396 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.548477 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.548955 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.548958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.548960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:05.549032 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.549034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.550099 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.550100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.551140 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.551142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.552109 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.552110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.552197 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.552214 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.552289 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.552345 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":109,"threads_started":1}
I20260927 11:18:05.552559 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.548201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37831 (local address 127.0.0.1:43356)
0927 11:18:05.548690 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.548702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.548776 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.549194 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.549195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.549198 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:05.549258 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.549260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.550700 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.550701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.551590 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.551592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.552300 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.552301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.552369 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.552373 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.552441 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.552483 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":24,"thread_start_us":116,"threads_started":1}
I20260927 11:18:05.552623 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.548513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.0.98.1:37831)
0927 11:18:05.548677 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.548682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.548760 (+ 78us) server_negotiation.cc:407] Connection header received
0927 11:18:05.548855 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.548858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.548875 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.549180 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:05.549348 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.550618 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.551679 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.552225 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.552299 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.552370 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.552456 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.552500 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260927 11:18:05.552901 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.548518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.0.98.3:38629)
0927 11:18:05.548769 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.548774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.548786 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:18:05.548831 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.548834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.548851 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.548946 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:05.549259 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.550007 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.551228 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.552024 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.552101 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.552195 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.552745 (+ 550us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.552790 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
I20260927 11:18:05.553371 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2211db571c4f94adc5da5c42610724 (DEFAULT_TABLE table=test-table [id=fe80fe94c74b42ef96e6f4cc63f22b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:05.553551 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2211db571c4f94adc5da5c42610724 (DEFAULT_TABLE table=test-table [id=fe80fe94c74b42ef96e6f4cc63f22b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:05.553762 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2211db571c4f94adc5da5c42610724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:05.553818 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.548201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44493 (local address 127.0.0.1:49748)
0927 11:18:05.548853 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.548862 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.548903 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.549268 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.549269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.549271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:05.549312 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.549314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.550713 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.550714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.551983 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.551987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.553549 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.553550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.553617 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.553622 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.553695 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.553738 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":4,"thread_start_us":109,"threads_started":1}
I20260927 11:18:05.553843 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2211db571c4f94adc5da5c42610724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:05.553853 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.548525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49748 (local address 127.0.98.2:44493)
0927 11:18:05.549107 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.549112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.549121 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:05.549157 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.549160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.549175 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.549263 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:05.549389 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.550591 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.552099 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.553463 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.553549 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.553637 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.553709 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.553747 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":124,"threads_started":1}
I20260927 11:18:05.554518 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2211db571c4f94adc5da5c42610724 (DEFAULT_TABLE table=test-table [id=fe80fe94c74b42ef96e6f4cc63f22b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:05.554781 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2211db571c4f94adc5da5c42610724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:05.557551 2015 tablet_bootstrap.cc:492] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2: Bootstrap starting.
I20260927 11:18:05.557945 2016 tablet_bootstrap.cc:492] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2: Bootstrap starting.
I20260927 11:18:05.558281 2015 tablet_bootstrap.cc:654] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:05.558568 2016 tablet_bootstrap.cc:654] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:05.558584 2015 log.cc:824] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:05.558894 2016 log.cc:824] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:05.559249 2017 tablet_bootstrap.cc:492] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949: Bootstrap starting.
I20260927 11:18:05.559304 2015 tablet_bootstrap.cc:492] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2: No bootstrap required, opened a new log
I20260927 11:18:05.559362 2015 ts_tablet_manager.cc:1397] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:05.559638 2016 tablet_bootstrap.cc:492] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2: No bootstrap required, opened a new log
I20260927 11:18:05.559703 2016 ts_tablet_manager.cc:1397] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:05.561388 2015 raft_consensus.cc:348] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.561571 2015 raft_consensus.cc:374] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:05.561619 2015 raft_consensus.cc:729] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee68b3a253ba4f4bbc98538e18f655a2, State: Initialized, Role: FOLLOWER
I20260927 11:18:05.561636 2017 tablet_bootstrap.cc:654] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:05.561667 2016 raft_consensus.cc:348] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.561766 2015 consensus_queue.cc:260] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.561815 2016 raft_consensus.cc:374] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:05.561847 2016 raft_consensus.cc:729] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952d92ec4b9f4188b3d6c690bb5feab2, State: Initialized, Role: FOLLOWER
I20260927 11:18:05.561980 2017 log.cc:824] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:05.561985 2016 consensus_queue.cc:260] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.562033 2015 ts_tablet_manager.cc:1428] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:18:05.562146 1991 heartbeater.cc:503] Master 127.0.98.62:40459 was elected leader, sending a full tablet report...
I20260927 11:18:05.562240 2016 ts_tablet_manager.cc:1428] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:18:05.562351 1711 heartbeater.cc:503] Master 127.0.98.62:40459 was elected leader, sending a full tablet report...
I20260927 11:18:05.562853 2017 tablet_bootstrap.cc:492] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949: No bootstrap required, opened a new log
I20260927 11:18:05.562927 2017 ts_tablet_manager.cc:1397] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 11:18:05.564832 2017 raft_consensus.cc:348] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.564997 2017 raft_consensus.cc:374] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:05.565063 2017 raft_consensus.cc:729] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae9c409c0d94ab7bc734bc4fbdaa949, State: Initialized, Role: FOLLOWER
I20260927 11:18:05.565186 2017 consensus_queue.cc:260] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.565452 2017 ts_tablet_manager.cc:1428] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 11:18:05.565522 1851 heartbeater.cc:503] Master 127.0.98.62:40459 was elected leader, sending a full tablet report...
W20260927 11:18:05.576566 1852 tablet.cc:2369] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:18:05.667361 1712 tablet.cc:2369] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:18:05.752480 1992 tablet.cc:2369] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:05.769227 2021 raft_consensus.cc:482] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:05.769349 2021 raft_consensus.cc:504] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.769694 2021 leader_election.cc:288] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bae9c409c0d94ab7bc734bc4fbdaa949 (127.0.98.2:44493), ee68b3a253ba4f4bbc98538e18f655a2 (127.0.98.3:38629)
I20260927 11:18:05.772754 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.769958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39363 (local address 127.0.98.2:44493)
0927 11:18:05.770132 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.770134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.770196 (+ 62us) server_negotiation.cc:407] Connection header received
0927 11:18:05.770255 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.770256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.770269 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.770349 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:05.770503 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.770974 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.772062 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.772439 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.772516 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.772526 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.772653 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.772686 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260927 11:18:05.772766 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.769886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44493 (local address 127.0.98.1:39363)
0927 11:18:05.770101 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.770114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.770199 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.770356 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.770357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.770359 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:05.770428 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.770429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.771087 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.771088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.771975 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.771977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.772516 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.772517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.772582 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.772585 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.772645 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.772683 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260927 11:18:05.772827 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.770151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45259 (local address 127.0.98.3:38629)
0927 11:18:05.770199 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.770201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.770252 (+ 51us) server_negotiation.cc:407] Connection header received
0927 11:18:05.770335 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.770336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.770347 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.770413 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:05.770551 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.770996 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.772067 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.772471 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.772558 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.772567 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.772726 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.772759 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:18:05.773089 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.769887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38629 (local address 127.0.98.1:45259)
0927 11:18:05.770193 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.770204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.770270 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.770415 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.770416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.770418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:05.770479 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.770481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.771096 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.771097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.771979 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.771981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.772559 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.772560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.772640 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.772648 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.772716 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.772755 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":65,"thread_start_us":150,"threads_started":1}
I20260927 11:18:05.773255 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2211db571c4f94adc5da5c42610724" candidate_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" is_pre_election: true
I20260927 11:18:05.773429 1805 raft_consensus.cc:2454] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 952d92ec4b9f4188b3d6c690bb5feab2 in term 0.
I20260927 11:18:05.773536 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2211db571c4f94adc5da5c42610724" candidate_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" is_pre_election: true
I20260927 11:18:05.773681 1945 raft_consensus.cc:2454] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 952d92ec4b9f4188b3d6c690bb5feab2 in term 0.
I20260927 11:18:05.773669 1596 leader_election.cc:302] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 952d92ec4b9f4188b3d6c690bb5feab2, bae9c409c0d94ab7bc734bc4fbdaa949; no voters:
I20260927 11:18:05.773806 2021 raft_consensus.cc:2781] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:05.773859 2021 raft_consensus.cc:482] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:05.773874 2021 raft_consensus.cc:3037] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:05.774603 2021 raft_consensus.cc:504] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.774746 2021 leader_election.cc:288] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [CANDIDATE]: Term 1 election: Requested vote from peers bae9c409c0d94ab7bc734bc4fbdaa949 (127.0.98.2:44493), ee68b3a253ba4f4bbc98538e18f655a2 (127.0.98.3:38629)
I20260927 11:18:05.774899 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2211db571c4f94adc5da5c42610724" candidate_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949"
I20260927 11:18:05.774974 1805 raft_consensus.cc:3037] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:05.774935 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2211db571c4f94adc5da5c42610724" candidate_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee68b3a253ba4f4bbc98538e18f655a2"
I20260927 11:18:05.775034 1945 raft_consensus.cc:3037] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:05.775717 1805 raft_consensus.cc:2454] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952d92ec4b9f4188b3d6c690bb5feab2 in term 1.
I20260927 11:18:05.775799 1945 raft_consensus.cc:2454] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952d92ec4b9f4188b3d6c690bb5feab2 in term 1.
I20260927 11:18:05.775875 1596 leader_election.cc:302] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 952d92ec4b9f4188b3d6c690bb5feab2, bae9c409c0d94ab7bc734bc4fbdaa949; no voters:
I20260927 11:18:05.775957 2021 raft_consensus.cc:2781] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:05.776077 2021 raft_consensus.cc:686] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [term 1 LEADER]: Becoming Leader. State: Replica: 952d92ec4b9f4188b3d6c690bb5feab2, State: Running, Role: LEADER
I20260927 11:18:05.776214 2021 consensus_queue.cc:237] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.776990 1507 catalog_manager.cc:5495] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 952d92ec4b9f4188b3d6c690bb5feab2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20260927 11:18:05.834805 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.832337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.0.98.1:37831)
0927 11:18:05.832398 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.832400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.832409 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:05.832498 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.832499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.832518 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.832576 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:05.832724 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.833274 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.834068 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.834254 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.834310 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.834473 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:05.834565 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:05.834620 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.834678 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.834695 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 11:18:05.836553 1945 raft_consensus.cc:1261] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2 [term 1 FOLLOWER]: Refusing update from remote peer 952d92ec4b9f4188b3d6c690bb5feab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:05.836587 1805 raft_consensus.cc:1261] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949 [term 1 FOLLOWER]: Refusing update from remote peer 952d92ec4b9f4188b3d6c690bb5feab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:05.836908 2021 consensus_queue.cc:1035] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:05.837061 2026 consensus_queue.cc:1035] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:05.840602 2031 mvcc.cc:205] Tried to move back new op lower bound from 7333920300383977472 to 7333920300140068864. Current Snapshot: MvccSnapshot[applied={T|T < 7333920300383977472 or (T in {7333920300383977472})}]
I20260927 11:18:05.842411 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:05.842509 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:05.843284 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795: Sending DeleteTablet for 3 replicas of tablet 7f2211db571c4f94adc5da5c42610724
I20260927 11:18:05.843627 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2211db571c4f94adc5da5c42610724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43356
I20260927 11:18:05.843756 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2211db571c4f94adc5da5c42610724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48666
I20260927 11:18:05.843853 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2211db571c4f94adc5da5c42610724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49748
I20260927 11:18:05.844036 2040 tablet_replica.cc:330] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949: stopping tablet replica
I20260927 11:18:05.844050 2038 tablet_replica.cc:330] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2: stopping tablet replica
I20260927 11:18:05.844144 2040 raft_consensus.cc:2229] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:05.844143 2038 raft_consensus.cc:2229] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:05.844245 2040 raft_consensus.cc:2258] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:05.844252 2038 raft_consensus.cc:2258] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:05.844381 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:05.844524 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:05.844702 2040 ts_tablet_manager.cc:1905] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:05.844992 2038 ts_tablet_manager.cc:1905] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:05.846086 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 89bb36b4ed5341279d1cde282386997d (DEFAULT_TABLE table=test-table [id=4a6cd1be66eb4c409f993187f19e78a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:05.846186 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89bb36b4ed5341279d1cde282386997d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:05.846657 2040 ts_tablet_manager.cc:1918] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:05.846735 2040 log.cc:1196] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/7f2211db571c4f94adc5da5c42610724
I20260927 11:18:05.846863 2038 ts_tablet_manager.cc:1918] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:05.846933 2038 log.cc:1196] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/wal/wals/7f2211db571c4f94adc5da5c42610724
I20260927 11:18:05.846984 2040 ts_tablet_manager.cc:1939] T 7f2211db571c4f94adc5da5c42610724 P bae9c409c0d94ab7bc734bc4fbdaa949: Deleting consensus metadata
I20260927 11:18:05.847234 2038 ts_tablet_manager.cc:1939] T 7f2211db571c4f94adc5da5c42610724 P 952d92ec4b9f4188b3d6c690bb5feab2: Deleting consensus metadata
I20260927 11:18:05.847365 1492 catalog_manager.cc:4841] TS bae9c409c0d94ab7bc734bc4fbdaa949 (127.0.98.2:44493): tablet 7f2211db571c4f94adc5da5c42610724 (table test-table [id=fe80fe94c74b42ef96e6f4cc63f22b54]) successfully deleted
I20260927 11:18:05.847633 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 89bb36b4ed5341279d1cde282386997d (DEFAULT_TABLE table=test-table [id=4a6cd1be66eb4c409f993187f19e78a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:05.847711 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89bb36b4ed5341279d1cde282386997d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:05.848464 2017 tablet_bootstrap.cc:492] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949: Bootstrap starting.
I20260927 11:18:05.848496 2016 tablet_bootstrap.cc:492] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2: Bootstrap starting.
I20260927 11:18:05.849261 2017 tablet_bootstrap.cc:654] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:05.849362 2016 tablet_bootstrap.cc:654] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:05.849495 2039 tablet_replica.cc:330] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2: stopping tablet replica
I20260927 11:18:05.849643 2039 raft_consensus.cc:2229] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:05.849669 1491 catalog_manager.cc:4841] TS 952d92ec4b9f4188b3d6c690bb5feab2 (127.0.98.1:37831): tablet 7f2211db571c4f94adc5da5c42610724 (table test-table [id=fe80fe94c74b42ef96e6f4cc63f22b54]) successfully deleted
I20260927 11:18:05.849810 2017 tablet_bootstrap.cc:492] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949: No bootstrap required, opened a new log
I20260927 11:18:05.849865 2017 ts_tablet_manager.cc:1397] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:05.849903 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 89bb36b4ed5341279d1cde282386997d (DEFAULT_TABLE table=test-table [id=4a6cd1be66eb4c409f993187f19e78a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:05.850023 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89bb36b4ed5341279d1cde282386997d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:05.850132 2039 raft_consensus.cc:2258] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:05.850207 2017 raft_consensus.cc:348] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.850277 2017 raft_consensus.cc:374] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:05.850296 2017 raft_consensus.cc:729] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae9c409c0d94ab7bc734bc4fbdaa949, State: Initialized, Role: FOLLOWER
I20260927 11:18:05.850338 2017 consensus_queue.cc:260] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.850438 2017 ts_tablet_manager.cc:1428] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:05.850788 2039 ts_tablet_manager.cc:1905] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:05.851358 2015 tablet_bootstrap.cc:492] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2: Bootstrap starting.
I20260927 11:18:05.851740 2015 tablet_bootstrap.cc:654] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:05.852067 2016 tablet_bootstrap.cc:492] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2: No bootstrap required, opened a new log
I20260927 11:18:05.852109 2016 ts_tablet_manager.cc:1397] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 11:18:05.852209 2016 raft_consensus.cc:348] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.852263 2016 raft_consensus.cc:374] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:05.852277 2016 raft_consensus.cc:729] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952d92ec4b9f4188b3d6c690bb5feab2, State: Initialized, Role: FOLLOWER
I20260927 11:18:05.852334 2016 consensus_queue.cc:260] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.852448 2016 ts_tablet_manager.cc:1428] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:05.853566 2039 ts_tablet_manager.cc:1918] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:05.853646 2039 log.cc:1196] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/7f2211db571c4f94adc5da5c42610724
I20260927 11:18:05.853863 2039 ts_tablet_manager.cc:1939] T 7f2211db571c4f94adc5da5c42610724 P ee68b3a253ba4f4bbc98538e18f655a2: Deleting consensus metadata
I20260927 11:18:05.853930 2015 tablet_bootstrap.cc:492] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2: No bootstrap required, opened a new log
I20260927 11:18:05.853979 2015 ts_tablet_manager.cc:1397] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 11:18:05.854139 2015 raft_consensus.cc:348] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.854205 2015 raft_consensus.cc:374] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:05.854228 2015 raft_consensus.cc:729] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee68b3a253ba4f4bbc98538e18f655a2, State: Initialized, Role: FOLLOWER
I20260927 11:18:05.854283 2015 consensus_queue.cc:260] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.854377 2015 ts_tablet_manager.cc:1428] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:05.854502 1493 catalog_manager.cc:4841] TS ee68b3a253ba4f4bbc98538e18f655a2 (127.0.98.3:38629): tablet 7f2211db571c4f94adc5da5c42610724 (table test-table [id=fe80fe94c74b42ef96e6f4cc63f22b54]) successfully deleted
I20260927 11:18:05.957013 2023 raft_consensus.cc:482] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:05.957185 2023 raft_consensus.cc:504] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.957531 2023 leader_election.cc:288] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 952d92ec4b9f4188b3d6c690bb5feab2 (127.0.98.1:37831), ee68b3a253ba4f4bbc98538e18f655a2 (127.0.98.3:38629)
I20260927 11:18:05.961985 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.957981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60707 (local address 127.0.98.1:37831)
0927 11:18:05.958021 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.958023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.958083 (+ 60us) server_negotiation.cc:407] Connection header received
0927 11:18:05.958178 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.958179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.958189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.958270 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:05.958488 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.959238 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.960976 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.961552 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.961631 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.961644 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.961872 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.961911 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:18:05.961985 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.957900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58823 (local address 127.0.98.3:38629)
0927 11:18:05.957943 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.957945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.958024 (+ 79us) server_negotiation.cc:407] Connection header received
0927 11:18:05.958195 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.958196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.958209 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.958270 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:05.958487 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.959232 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.960976 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.961552 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.961635 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.961644 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.961872 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.961911 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:18:05.962085 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.957779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38629 (local address 127.0.98.2:58823)
0927 11:18:05.957942 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.957956 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.958090 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.958293 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.958294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.958296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:05.958408 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.958411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.959338 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.959339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.960857 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.960860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.961683 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.961684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.961777 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.961785 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.961858 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.961921 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":54,"thread_start_us":93,"threads_started":1}
I20260927 11:18:05.962260 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.957779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37831 (local address 127.0.98.2:60707)
0927 11:18:05.958021 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.958032 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.958091 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.958293 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.958294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.958298 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:05.958411 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.958412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.959338 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.959339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.960857 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.960860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.961683 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.961684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.961777 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.961787 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.961858 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.961921 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":150,"thread_start_us":92,"threads_started":1}
I20260927 11:18:05.962404 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bb36b4ed5341279d1cde282386997d" candidate_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" is_pre_election: true
I20260927 11:18:05.962509 1945 raft_consensus.cc:2454] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae9c409c0d94ab7bc734bc4fbdaa949 in term 0.
I20260927 11:18:05.962527 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bb36b4ed5341279d1cde282386997d" candidate_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" is_pre_election: true
I20260927 11:18:05.962662 1665 raft_consensus.cc:2454] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae9c409c0d94ab7bc734bc4fbdaa949 in term 0.
I20260927 11:18:05.962756 1738 leader_election.cc:302] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bae9c409c0d94ab7bc734bc4fbdaa949, ee68b3a253ba4f4bbc98538e18f655a2; no voters:
I20260927 11:18:05.962921 2023 raft_consensus.cc:2781] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:05.962973 2023 raft_consensus.cc:482] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:05.962990 2023 raft_consensus.cc:3037] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:05.964128 2023 raft_consensus.cc:504] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.964324 2023 leader_election.cc:288] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [CANDIDATE]: Term 1 election: Requested vote from peers 952d92ec4b9f4188b3d6c690bb5feab2 (127.0.98.1:37831), ee68b3a253ba4f4bbc98538e18f655a2 (127.0.98.3:38629)
I20260927 11:18:05.964524 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bb36b4ed5341279d1cde282386997d" candidate_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952d92ec4b9f4188b3d6c690bb5feab2"
I20260927 11:18:05.964519 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bb36b4ed5341279d1cde282386997d" candidate_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee68b3a253ba4f4bbc98538e18f655a2"
I20260927 11:18:05.964610 1665 raft_consensus.cc:3037] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:05.964612 1945 raft_consensus.cc:3037] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:05.965397 1945 raft_consensus.cc:2454] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae9c409c0d94ab7bc734bc4fbdaa949 in term 1.
I20260927 11:18:05.965412 1665 raft_consensus.cc:2454] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae9c409c0d94ab7bc734bc4fbdaa949 in term 1.
I20260927 11:18:05.965627 1736 leader_election.cc:302] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 952d92ec4b9f4188b3d6c690bb5feab2, bae9c409c0d94ab7bc734bc4fbdaa949; no voters:
I20260927 11:18:05.965797 2023 raft_consensus.cc:2781] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:05.965963 2023 raft_consensus.cc:686] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [term 1 LEADER]: Becoming Leader. State: Replica: bae9c409c0d94ab7bc734bc4fbdaa949, State: Running, Role: LEADER
I20260927 11:18:05.966053 2023 consensus_queue.cc:237] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:05.966790 1505 catalog_manager.cc:5495] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 reported cstate change: term changed from 0 to 1, leader changed from <none> to bae9c409c0d94ab7bc734bc4fbdaa949 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } health_report { overall_health: UNKNOWN } } }
Sep 27 11:18:05 dist-test-slave-lsvg krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:05.998605 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.985785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40459 (local address 127.0.0.1:35172)
0927 11:18:05.985978 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:05.985989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:05.986076 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:05.986257 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:05.986258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:05.986574 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:05.986620 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.986623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.987276 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.987278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.987799 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:05.987801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.988146 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.988148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:05.988202 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.994291 (+ 6089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:05.994297 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:05.995842 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:05.997661 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.997670 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.997672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.997890 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.998066 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:05.998067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:05.998067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:05.998126 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:05.998331 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:05.998337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:05.998382 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 11:18:05.998435 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 11:18:05.998523 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":110,"threads_started":1}
I20260927 11:18:05.998724 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:05.985863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.0.98.62:40459)
0927 11:18:05.985947 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:05.985949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:05.986055 (+ 106us) server_negotiation.cc:407] Connection header received
0927 11:18:05.986170 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:05.986171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:05.986183 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:05.986253 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:05.986716 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.987202 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.987879 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:05.988058 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:05.988146 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:05.995981 (+ 7835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:05.995983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:05.995983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:05.995991 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:05.997558 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.997958 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.997959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.997959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.998012 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:05.998176 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:05.998177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:05.998177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:05.998283 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:05.998332 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:05.998512 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 11:18:05.998578 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 11:18:06.000789 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35172:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 11:18:06.001040 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:06.002568 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 1bee708a863d4b8980822519bfed966e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1fd6428191c4494a50d2252809d46ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:18:06.002604 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 1bee708a863d4b8980822519bfed966e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1fd6428191c4494a50d2252809d46ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:18:06.002619 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 1bee708a863d4b8980822519bfed966e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e1fd6428191c4494a50d2252809d46ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:18:06.002760 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bee708a863d4b8980822519bfed966e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:06.002761 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bee708a863d4b8980822519bfed966e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:06.002768 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bee708a863d4b8980822519bfed966e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:06.004179 2016 tablet_bootstrap.cc:492] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2: Bootstrap starting.
I20260927 11:18:06.004298 2017 tablet_bootstrap.cc:492] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949: Bootstrap starting.
I20260927 11:18:06.004289 2015 tablet_bootstrap.cc:492] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2: Bootstrap starting.
I20260927 11:18:06.004524 2016 tablet_bootstrap.cc:654] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:06.004712 2015 tablet_bootstrap.cc:654] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:06.004712 2017 tablet_bootstrap.cc:654] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:06.005081 2016 tablet_bootstrap.cc:492] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2: No bootstrap required, opened a new log
I20260927 11:18:06.005136 2016 ts_tablet_manager.cc:1397] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:06.005250 2016 raft_consensus.cc:348] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:06.005328 2016 raft_consensus.cc:374] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:06.005352 2016 raft_consensus.cc:729] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952d92ec4b9f4188b3d6c690bb5feab2, State: Initialized, Role: FOLLOWER
I20260927 11:18:06.005404 2016 consensus_queue.cc:260] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:06.005483 2017 tablet_bootstrap.cc:492] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949: No bootstrap required, opened a new log
I20260927 11:18:06.005517 2016 ts_tablet_manager.cc:1428] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:06.005530 2017 ts_tablet_manager.cc:1397] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 11:18:06.005611 2026 tablet_replica.cc:439] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } } }
I20260927 11:18:06.005677 2026 tablet_replica.cc:442] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:06.005715 2015 tablet_bootstrap.cc:492] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2: No bootstrap required, opened a new log
I20260927 11:18:06.005681 2017 raft_consensus.cc:348] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:06.005751 2017 raft_consensus.cc:374] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:06.005755 2015 ts_tablet_manager.cc:1397] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:06.005774 2017 raft_consensus.cc:729] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae9c409c0d94ab7bc734bc4fbdaa949, State: Initialized, Role: FOLLOWER
I20260927 11:18:06.005820 2017 consensus_queue.cc:260] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:06.005913 2017 ts_tablet_manager.cc:1428] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:06.005895 2015 raft_consensus.cc:348] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:06.005956 2015 raft_consensus.cc:374] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:06.005995 2015 raft_consensus.cc:729] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee68b3a253ba4f4bbc98538e18f655a2, State: Initialized, Role: FOLLOWER
I20260927 11:18:06.006038 2015 consensus_queue.cc:260] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:06.006031 2023 tablet_replica.cc:439] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } } }
I20260927 11:18:06.006093 2023 tablet_replica.cc:442] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:06.006145 2015 ts_tablet_manager.cc:1428] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:06.006191 2020 tablet_replica.cc:439] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } } }
I20260927 11:18:06.006317 2020 tablet_replica.cc:442] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:06.068650 2023 raft_consensus.cc:482] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:06.068742 2023 raft_consensus.cc:504] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:06.068895 2023 leader_election.cc:288] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 952d92ec4b9f4188b3d6c690bb5feab2 (127.0.98.1:37831), ee68b3a253ba4f4bbc98538e18f655a2 (127.0.98.3:38629)
I20260927 11:18:06.069157 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bee708a863d4b8980822519bfed966e" candidate_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" is_pre_election: true
I20260927 11:18:06.069295 1945 raft_consensus.cc:2454] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae9c409c0d94ab7bc734bc4fbdaa949 in term 0.
I20260927 11:18:06.069200 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bee708a863d4b8980822519bfed966e" candidate_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" is_pre_election: true
I20260927 11:18:06.069345 1665 raft_consensus.cc:2454] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bae9c409c0d94ab7bc734bc4fbdaa949 in term 0.
I20260927 11:18:06.069454 1738 leader_election.cc:302] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bae9c409c0d94ab7bc734bc4fbdaa949, ee68b3a253ba4f4bbc98538e18f655a2; no voters:
I20260927 11:18:06.069569 2023 raft_consensus.cc:2781] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:06.069621 2023 raft_consensus.cc:482] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:06.069643 2023 raft_consensus.cc:3037] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:06.070453 2023 raft_consensus.cc:504] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:06.070582 2023 leader_election.cc:288] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [CANDIDATE]: Term 1 election: Requested vote from peers 952d92ec4b9f4188b3d6c690bb5feab2 (127.0.98.1:37831), ee68b3a253ba4f4bbc98538e18f655a2 (127.0.98.3:38629)
I20260927 11:18:06.070781 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bee708a863d4b8980822519bfed966e" candidate_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee68b3a253ba4f4bbc98538e18f655a2"
I20260927 11:18:06.070796 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bee708a863d4b8980822519bfed966e" candidate_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952d92ec4b9f4188b3d6c690bb5feab2"
I20260927 11:18:06.070866 1945 raft_consensus.cc:3037] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:06.070868 1665 raft_consensus.cc:3037] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:06.071615 1945 raft_consensus.cc:2454] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae9c409c0d94ab7bc734bc4fbdaa949 in term 1.
I20260927 11:18:06.071609 1665 raft_consensus.cc:2454] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bae9c409c0d94ab7bc734bc4fbdaa949 in term 1.
I20260927 11:18:06.071805 1738 leader_election.cc:302] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bae9c409c0d94ab7bc734bc4fbdaa949, ee68b3a253ba4f4bbc98538e18f655a2; no voters:
I20260927 11:18:06.071900 2023 raft_consensus.cc:2781] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:06.071955 2023 raft_consensus.cc:686] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [term 1 LEADER]: Becoming Leader. State: Replica: bae9c409c0d94ab7bc734bc4fbdaa949, State: Running, Role: LEADER
I20260927 11:18:06.072005 2023 consensus_queue.cc:237] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } }
I20260927 11:18:06.072367 2046 tablet_replica.cc:439] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949: TxnStatusTablet state changed. Reason: New leader bae9c409c0d94ab7bc734bc4fbdaa949. Latest consensus state: current_term: 1 leader_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } } }
I20260927 11:18:06.072489 2046 tablet_replica.cc:442] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:18:06.072718 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 11:18:06.072721 1506 catalog_manager.cc:5495] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 reported cstate change: term changed from 0 to 1, leader changed from <none> to bae9c409c0d94ab7bc734bc4fbdaa949 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20260927 11:18:06.072844 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 11:18:06.080972 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:06.078492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44493 (local address 127.0.0.1:49756)
0927 11:18:06.078565 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:06.078577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:06.078639 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:06.078788 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:06.078789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:06.078790 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:18:06.078848 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:06.078850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.079480 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.079481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:06.080059 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:06.080061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.080423 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.080424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:06.080482 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:06.080499 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:18:06.080790 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:18:06.080800 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 11:18:06.080845 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 11:18:06.080903 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 11:18:06.080955 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:06.078566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49756 (local address 127.0.98.2:44493)
0927 11:18:06.078612 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:06.078614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:06.078624 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:06.078708 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:06.078709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:06.078728 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:06.078786 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:06.078906 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.079398 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.080142 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.080352 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.080426 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:06.080604 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:06.080724 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:06.080803 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:06.080858 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 11:18:06.080879 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:18:06.082202 1945 raft_consensus.cc:1261] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2 [term 1 FOLLOWER]: Refusing update from remote peer bae9c409c0d94ab7bc734bc4fbdaa949: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:06.082260 1665 raft_consensus.cc:1261] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2 [term 1 FOLLOWER]: Refusing update from remote peer bae9c409c0d94ab7bc734bc4fbdaa949: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:06.082464 2046 consensus_queue.cc:1035] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:06.082607 2023 consensus_queue.cc:1035] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:06.083197 2026 tablet_replica.cc:439] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2: TxnStatusTablet state changed. Reason: New leader bae9c409c0d94ab7bc734bc4fbdaa949. Latest consensus state: current_term: 1 leader_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } } }
I20260927 11:18:06.083312 2026 tablet_replica.cc:442] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:06.083473 2020 tablet_replica.cc:439] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2: TxnStatusTablet state changed. Reason: New leader bae9c409c0d94ab7bc734bc4fbdaa949. Latest consensus state: current_term: 1 leader_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } } }
I20260927 11:18:06.083580 2020 tablet_replica.cc:442] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:06.084650 2046 tablet_replica.cc:439] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } } }
I20260927 11:18:06.084740 2046 tablet_replica.cc:442] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:18:06.084738 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333920301390733312 to 7333920301351473152. Current Snapshot: MvccSnapshot[applied={T|T < 7333920301390733312}]
I20260927 11:18:06.084790 2046 tablet_replica.cc:439] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } } }
I20260927 11:18:06.084833 2046 tablet_replica.cc:442] T 1bee708a863d4b8980822519bfed966e P bae9c409c0d94ab7bc734bc4fbdaa949: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:18:06.084998 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333920301390733312 to 7333920301351473152. Current Snapshot: MvccSnapshot[applied={T|T < 7333920301390733312}]
I20260927 11:18:06.085230 2026 tablet_replica.cc:439] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } } }
I20260927 11:18:06.085310 2026 tablet_replica.cc:442] T 1bee708a863d4b8980822519bfed966e P 952d92ec4b9f4188b3d6c690bb5feab2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:06.085690 2020 tablet_replica.cc:439] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 } } peers { permanent_uuid: "bae9c409c0d94ab7bc734bc4fbdaa949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44493 } } peers { permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 } } }
I20260927 11:18:06.085785 2020 tablet_replica.cc:442] T 1bee708a863d4b8980822519bfed966e P ee68b3a253ba4f4bbc98538e18f655a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:06.089783 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:06.087223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49760 (local address 127.0.98.2:44493)
0927 11:18:06.087276 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:06.087279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:06.087287 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:06.087310 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:06.087311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:06.087322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:06.087387 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:06.087474 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.088286 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.088969 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.089323 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.089401 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:06.089547 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:06.089582 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:06.089632 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:06.089685 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 11:18:06.089706 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:18:06.095105 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:06.092270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53599 (local address 127.0.98.2:44493)
0927 11:18:06.092312 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:06.092313 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:06.092450 (+ 137us) server_negotiation.cc:407] Connection header received
0927 11:18:06.092500 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:06.092501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:06.092508 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:06.092552 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:06.092691 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.093168 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.094435 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.094610 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.094655 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:06.094860 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:06.094881 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:06.094919 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:06.094995 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 11:18:06.095035 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":22}
I20260927 11:18:06.095208 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:06.092217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44493 (local address 127.0.98.2:53599)
0927 11:18:06.092383 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:06.092394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:06.092452 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:06.092556 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:06.092557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:06.092558 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:18:06.092630 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:06.092632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.093235 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.093236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:06.094343 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:06.094346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.094696 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.094697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:06.094766 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:06.094782 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:18:06.094935 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:18:06.094936 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:18:06.094992 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 11:18:06.095060 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":5,"thread_start_us":79,"threads_started":1}
I20260927 11:18:06.097414 1945 raft_consensus.cc:1261] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2 [term 1 FOLLOWER]: Refusing update from remote peer bae9c409c0d94ab7bc734bc4fbdaa949: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:06.097568 1665 raft_consensus.cc:1261] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2 [term 1 FOLLOWER]: Refusing update from remote peer bae9c409c0d94ab7bc734bc4fbdaa949: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:06.097723 2023 consensus_queue.cc:1035] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee68b3a253ba4f4bbc98538e18f655a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:06.097834 2046 consensus_queue.cc:1035] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "952d92ec4b9f4188b3d6c690bb5feab2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:06.111420 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:06.111544 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:06.112159 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bc08646ced84351ae8ced0376aa8795: Sending DeleteTablet for 3 replicas of tablet 89bb36b4ed5341279d1cde282386997d
I20260927 11:18:06.112407 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 89bb36b4ed5341279d1cde282386997d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43356
I20260927 11:18:06.112409 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 89bb36b4ed5341279d1cde282386997d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49748
I20260927 11:18:06.112504 2038 tablet_replica.cc:330] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2: stopping tablet replica
I20260927 11:18:06.112515 2040 tablet_replica.cc:330] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949: stopping tablet replica
I20260927 11:18:06.112531 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 89bb36b4ed5341279d1cde282386997d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48666
I20260927 11:18:06.112556 2038 raft_consensus.cc:2229] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:06.112572 2040 raft_consensus.cc:2229] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:06.112619 2038 raft_consensus.cc:2258] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:06.112634 2039 tablet_replica.cc:330] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2: stopping tablet replica
I20260927 11:18:06.112673 2040 raft_consensus.cc:2258] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:06.112689 2039 raft_consensus.cc:2229] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:06.112725 2039 raft_consensus.cc:2258] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:06.113004 2038 ts_tablet_manager.cc:1905] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:06.113004 2040 ts_tablet_manager.cc:1905] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:06.113029 2039 ts_tablet_manager.cc:1905] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:06.114768 2039 ts_tablet_manager.cc:1918] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:06.114797 2040 ts_tablet_manager.cc:1918] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:06.114796 2038 ts_tablet_manager.cc:1918] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:06.114827 2039 log.cc:1196] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/89bb36b4ed5341279d1cde282386997d
I20260927 11:18:06.114847 2040 log.cc:1196] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/89bb36b4ed5341279d1cde282386997d
I20260927 11:18:06.114852 2038 log.cc:1196] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507878147783-392-0/minicluster-data/ts-0/wal/wals/89bb36b4ed5341279d1cde282386997d
I20260927 11:18:06.115060 2039 ts_tablet_manager.cc:1939] T 89bb36b4ed5341279d1cde282386997d P ee68b3a253ba4f4bbc98538e18f655a2: Deleting consensus metadata
I20260927 11:18:06.115060 2040 ts_tablet_manager.cc:1939] T 89bb36b4ed5341279d1cde282386997d P bae9c409c0d94ab7bc734bc4fbdaa949: Deleting consensus metadata
I20260927 11:18:06.115090 2038 ts_tablet_manager.cc:1939] T 89bb36b4ed5341279d1cde282386997d P 952d92ec4b9f4188b3d6c690bb5feab2: Deleting consensus metadata
I20260927 11:18:06.115377 1493 catalog_manager.cc:4841] TS ee68b3a253ba4f4bbc98538e18f655a2 (127.0.98.3:38629): tablet 89bb36b4ed5341279d1cde282386997d (table test-table [id=4a6cd1be66eb4c409f993187f19e78a2]) successfully deleted
I20260927 11:18:06.115459 1491 catalog_manager.cc:4841] TS 952d92ec4b9f4188b3d6c690bb5feab2 (127.0.98.1:37831): tablet 89bb36b4ed5341279d1cde282386997d (table test-table [id=4a6cd1be66eb4c409f993187f19e78a2]) successfully deleted
I20260927 11:18:06.115517 1492 catalog_manager.cc:4841] TS bae9c409c0d94ab7bc734bc4fbdaa949 (127.0.98.2:44493): tablet 89bb36b4ed5341279d1cde282386997d (table test-table [id=4a6cd1be66eb4c409f993187f19e78a2]) successfully deleted
Sep 27 11:18:06 dist-test-slave-lsvg krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:18:06.126747 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:06.117330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.0.98.1:37831)
0927 11:18:06.117377 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:06.117379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:06.117518 (+ 139us) server_negotiation.cc:407] Connection header received
0927 11:18:06.117614 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:06.117615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:06.117626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:06.117673 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:06.118398 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.118866 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.120030 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.120218 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.120282 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:06.123806 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:06.123817 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:06.123818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:06.123826 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:06.125539 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:06.125932 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:06.125936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:06.125937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:06.125989 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:06.126187 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:06.126188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:06.126189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:06.126338 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:06.126509 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:06.126546 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 11:18:06.126634 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 11:18:06.127436 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43374
I20260927 11:18:06.135682 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:06.128473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35184 (local address 127.0.98.62:40459)
0927 11:18:06.128550 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:06.128552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:06.128562 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:06.128640 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:06.128641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:06.128652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:06.128725 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:06.129389 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.130263 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.130975 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:06.131321 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:06.131401 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:06.132595 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:06.132596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:06.132597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:06.132606 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:06.134501 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:06.134885 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:06.134886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:06.134886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:06.134954 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:06.135127 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:06.135128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:06.135128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:06.135293 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:06.135404 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:06.135476 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 11:18:06.135571 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 11:18:06.135964 1505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35184
I20260927 11:18:06.137110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 1579
I20260927 11:18:06.143914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 1720
I20260927 11:18:06.149478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 1860
I20260927 11:18:06.154026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 1475
2026-09-27T11:18:06Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2998 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:06 dist-test-slave-lsvg krb5kdc[2076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:06 dist-test-slave-lsvg krb5kdc[2076](info): set up 2 sockets
Sep 27 11:18:06 dist-test-slave-lsvg krb5kdc[2076](info): commencing operation
krb5kdc: starting...
W20260927 11:18:07.224733 392 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:07 dist-test-slave-lsvg krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:18:07.344376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42137
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:07.424496 2089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:07.424759 2089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:07.424798 2089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:07.425966 2089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:07.426035 2089 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 11:18:07.426059 2089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:07.426090 2089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:07.426128 2089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:07.426158 2089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:07.427614 2089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42137
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:07.427904 2089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:07.428166 2089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:07.432951 2094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:07.433102 2089 server_base.cc:1034] running on GCE node
W20260927 11:18:07.432936 2097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:07.432940 2095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:07.433424 2089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:07.433743 2089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:07.434931 2089 hybrid_clock.cc:648] HybridClock initialized: now 1790507887434898 us; error 50 us; skew 500 ppm
Sep 27 11:18:07 dist-test-slave-lsvg krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:07.437762 2089 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:18:07.438213 2089 webserver.cc:460] Webserver started at http://127.0.98.62:43931/ using document root <none> and password file <none>
I20260927 11:18:07.438395 2089 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:07.438434 2089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:07.438546 2089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:07.439579 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "82b70d7bb3d841b28899e4dd6432be78"
format_stamp: "Formatted at 2026-09-27 11:18:07 on dist-test-slave-lsvg"
server_key: "a03ab8ae685b58e64b5bcf9dd274ba68"
server_key_iv: "88eee72e97105a90b3abeb14eaba2b98"
server_key_version: "encryptionkey@0"
I20260927 11:18:07.439945 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "82b70d7bb3d841b28899e4dd6432be78"
format_stamp: "Formatted at 2026-09-27 11:18:07 on dist-test-slave-lsvg"
server_key: "a03ab8ae685b58e64b5bcf9dd274ba68"
server_key_iv: "88eee72e97105a90b3abeb14eaba2b98"
server_key_version: "encryptionkey@0"
I20260927 11:18:07.441416 2089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:07.442194 2104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:07.442392 2089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:07.442479 2089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "82b70d7bb3d841b28899e4dd6432be78"
format_stamp: "Formatted at 2026-09-27 11:18:07 on dist-test-slave-lsvg"
server_key: "a03ab8ae685b58e64b5bcf9dd274ba68"
server_key_iv: "88eee72e97105a90b3abeb14eaba2b98"
server_key_version: "encryptionkey@0"
I20260927 11:18:07.442535 2089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:07.467705 2089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:07.470028 2089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:07.470186 2089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:07.474743 2089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42137
I20260927 11:18:07.474856 2166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42137 every 8 connection(s)
I20260927 11:18:07.475162 2089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:07.475822 2167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:07.478204 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78: Bootstrap starting.
I20260927 11:18:07.478781 2167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:07.479059 2167 log.cc:824] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:07.479090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 2089
I20260927 11:18:07.479206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:07.479378 392 external_mini_cluster.cc:1428] Setting key 8a109284427172cc6171e5b7f85e9042
I20260927 11:18:07.479708 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78: No bootstrap required, opened a new log
I20260927 11:18:07.481945 2167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b70d7bb3d841b28899e4dd6432be78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42137 } }
I20260927 11:18:07.482101 2167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:07.482164 2167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b70d7bb3d841b28899e4dd6432be78, State: Initialized, Role: FOLLOWER
I20260927 11:18:07.482301 2167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b70d7bb3d841b28899e4dd6432be78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42137 } }
I20260927 11:18:07.482395 2167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:07.482439 2167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:07.482496 2167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:07.483001 2167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b70d7bb3d841b28899e4dd6432be78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42137 } }
I20260927 11:18:07.483110 2167 leader_election.cc:302] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82b70d7bb3d841b28899e4dd6432be78; no voters:
I20260927 11:18:07.483318 2167 leader_election.cc:288] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:07.483351 2172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:07.483458 2172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [term 1 LEADER]: Becoming Leader. State: Replica: 82b70d7bb3d841b28899e4dd6432be78, State: Running, Role: LEADER
I20260927 11:18:07.483583 2167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:07.483594 2172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b70d7bb3d841b28899e4dd6432be78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42137 } }
I20260927 11:18:07.484045 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82b70d7bb3d841b28899e4dd6432be78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b70d7bb3d841b28899e4dd6432be78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42137 } } }
I20260927 11:18:07.484133 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:07.484357 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82b70d7bb3d841b28899e4dd6432be78. Latest consensus state: current_term: 1 leader_uuid: "82b70d7bb3d841b28899e4dd6432be78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b70d7bb3d841b28899e4dd6432be78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42137 } } }
I20260927 11:18:07.484584 2179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:07.484747 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:07.485292 2179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:07.487623 2179 catalog_manager.cc:1347] Generated new cluster ID: 4a254f1c3e094614890b8f91aa783e30
I20260927 11:18:07.487694 2179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:18:07 dist-test-slave-lsvg krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:07.492537 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.479847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37216 (local address 127.0.98.62:42137)
0927 11:18:07.480048 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:07.480061 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:07.480086 (+ 25us) server_negotiation.cc:407] Connection header received
0927 11:18:07.481958 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:07.481965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:07.482019 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:07.482186 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:07.482875 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.484070 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.485190 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.485745 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.485856 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.488747 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:07.488751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:07.488751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:07.488763 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:07.490799 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.491217 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.491220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.491220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.491346 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.491596 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.491597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.491597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.491901 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:07.492090 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:07.492218 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 11:18:07.492346 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
I20260927 11:18:07.501703 2179 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:07.502399 2179 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:07.508689 2179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78: Generated new TSK 0
I20260927 11:18:07.508886 2179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:18:07.540012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42137
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:07.621703 2193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:07.621906 2193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:07.621930 2193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:07.623059 2193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:07.623101 2193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:07.623122 2193 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 11:18:07.623134 2193 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:18:07.623173 2193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:07.624583 2193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:07.624816 2193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:07.625101 2193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:07.630051 2199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:07.630074 2201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:07.630188 2193 server_base.cc:1034] running on GCE node
W20260927 11:18:07.630051 2198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:07.630510 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:07.630760 2193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:07.631971 2193 hybrid_clock.cc:648] HybridClock initialized: now 1790507887631961 us; error 31 us; skew 500 ppm
Sep 27 11:18:07 dist-test-slave-lsvg krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:07.634590 2193 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:18:07.634991 2193 webserver.cc:460] Webserver started at http://127.0.98.1:37587/ using document root <none> and password file <none>
I20260927 11:18:07.635177 2193 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:07.635217 2193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:07.635305 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:07.636305 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "c7f08c903b69472c872448f912869947"
format_stamp: "Formatted at 2026-09-27 11:18:07 on dist-test-slave-lsvg"
server_key: "8b4dacafd1821d2295f9d51bf48cd203"
server_key_iv: "058f3833e27cf726cd7f8d60cbdf5cba"
server_key_version: "encryptionkey@0"
I20260927 11:18:07.636600 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c7f08c903b69472c872448f912869947"
format_stamp: "Formatted at 2026-09-27 11:18:07 on dist-test-slave-lsvg"
server_key: "8b4dacafd1821d2295f9d51bf48cd203"
server_key_iv: "058f3833e27cf726cd7f8d60cbdf5cba"
server_key_version: "encryptionkey@0"
I20260927 11:18:07.637917 2193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:07.638633 2208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:07.638801 2193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:07.638895 2193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "c7f08c903b69472c872448f912869947"
format_stamp: "Formatted at 2026-09-27 11:18:07 on dist-test-slave-lsvg"
server_key: "8b4dacafd1821d2295f9d51bf48cd203"
server_key_iv: "058f3833e27cf726cd7f8d60cbdf5cba"
server_key_version: "encryptionkey@0"
I20260927 11:18:07.638984 2193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:07.646922 2193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:07.649274 2193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:07.649425 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:07.649907 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:07.649971 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:07.650017 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:07.650048 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:07.656436 2193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40711
I20260927 11:18:07.656541 2323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40711 every 8 connection(s)
I20260927 11:18:07.656813 2193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:18:07 dist-test-slave-lsvg krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:07.660395 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.650405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40227 (local address 127.0.98.62:42137)
0927 11:18:07.650477 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:07.650481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:07.650716 (+ 235us) server_negotiation.cc:407] Connection header received
0927 11:18:07.651266 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:07.651268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:07.651289 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:07.651367 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:07.652233 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.653116 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.654666 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.654948 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.655039 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.657585 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:07.657586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:07.657587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:07.657596 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:07.659028 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.659452 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.659453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.659453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.659511 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.659746 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.659747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.659747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.659865 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:07.659931 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:07.660208 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 11:18:07.660296 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 11:18:07.660553 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.650445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42137 (local address 127.0.98.1:40227)
0927 11:18:07.650601 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:07.650641 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:07.651133 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:07.651450 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:07.651453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:07.651826 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:07.652123 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:07.652129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.653235 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.653237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:07.654578 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:07.654581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.655034 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.655035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:07.655195 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.655740 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:07.655752 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:07.657478 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:07.659128 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:07.659135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:07.659137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:07.659370 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:07.659572 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:07.659573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:07.659573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:07.659682 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:07.659933 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:07.659941 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:07.660074 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 11:18:07.660170 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 11:18:07.660326 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260927 11:18:07.661033 2324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42137
I20260927 11:18:07.661167 2324 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:07.661345 2324 heartbeater.cc:511] Master 127.0.98.62:42137 requested a full tablet report, sending...
I20260927 11:18:07.661875 2121 ts_manager.cc:194] Registered new tserver with Master: c7f08c903b69472c872448f912869947 (127.0.98.1:40711)
I20260927 11:18:07.662667 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40227
I20260927 11:18:07.664913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 2193
I20260927 11:18:07.665005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:07.665216 392 external_mini_cluster.cc:1428] Setting key a1678685fba83708bfd3ff31dea6f829
I20260927 11:18:07.667457 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.662095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42137 (local address 127.0.98.1:55081)
0927 11:18:07.662221 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:07.662231 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:07.662293 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:07.662424 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:07.662425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:07.662600 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:07.662647 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:07.662649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.663301 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.663302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:07.663840 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:07.663842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.664133 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.664133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:07.664184 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.664548 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:07.664552 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:07.664824 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:07.666466 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:07.666467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:07.666468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:07.666695 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:07.666862 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:07.666863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:07.666863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:07.666925 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:07.667215 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:07.667216 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:07.667250 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 11:18:07.667304 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 11:18:07.667380 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260927 11:18:07.667533 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.662148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55081 (local address 127.0.98.62:42137)
0927 11:18:07.662190 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:07.662192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:07.662289 (+ 97us) server_negotiation.cc:407] Connection header received
0927 11:18:07.662356 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:07.662356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:07.662366 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:07.662420 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:07.662709 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.663210 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.663906 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.664076 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.664125 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.664908 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:07.664909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:07.664910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:07.664917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:07.666383 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.666764 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.666764 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.666765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.666812 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.667000 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.667001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.667002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.667142 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:07.667213 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:07.667342 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 11:18:07.667424 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:18:07.691979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42137
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:07.772110 2333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:07.772326 2333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:07.772369 2333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:07.773564 2333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:07.773620 2333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:07.773658 2333 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 11:18:07.773707 2333 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:18:07.773753 2333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:07.775254 2333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:07.775507 2333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:07.775735 2333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:07.780658 2338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:07.780658 2341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:07.780875 2333 server_base.cc:1034] running on GCE node
W20260927 11:18:07.780750 2339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:07.781175 2333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:07.781478 2333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:07.782650 2333 hybrid_clock.cc:648] HybridClock initialized: now 1790507887782642 us; error 35 us; skew 500 ppm
Sep 27 11:18:07 dist-test-slave-lsvg krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:07.785281 2333 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:18:07.785705 2333 webserver.cc:460] Webserver started at http://127.0.98.2:37271/ using document root <none> and password file <none>
I20260927 11:18:07.785953 2333 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:07.786019 2333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:07.786154 2333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:07.787187 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "78c00bb8829b4553b75bcee4fcb25481"
format_stamp: "Formatted at 2026-09-27 11:18:07 on dist-test-slave-lsvg"
server_key: "6d933d0c24efe6532adb4bdf27d05b2a"
server_key_iv: "954d1b4fcbe8efc41dfee3f65ffe3ab2"
server_key_version: "encryptionkey@0"
I20260927 11:18:07.787542 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78c00bb8829b4553b75bcee4fcb25481"
format_stamp: "Formatted at 2026-09-27 11:18:07 on dist-test-slave-lsvg"
server_key: "6d933d0c24efe6532adb4bdf27d05b2a"
server_key_iv: "954d1b4fcbe8efc41dfee3f65ffe3ab2"
server_key_version: "encryptionkey@0"
I20260927 11:18:07.788928 2333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:07.789770 2348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:07.789949 2333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:07.790052 2333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "78c00bb8829b4553b75bcee4fcb25481"
format_stamp: "Formatted at 2026-09-27 11:18:07 on dist-test-slave-lsvg"
server_key: "6d933d0c24efe6532adb4bdf27d05b2a"
server_key_iv: "954d1b4fcbe8efc41dfee3f65ffe3ab2"
server_key_version: "encryptionkey@0"
I20260927 11:18:07.790124 2333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:07.805979 2333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:07.808101 2333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:07.808270 2333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:07.808780 2333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:07.808843 2333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:07.808887 2333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:07.808938 2333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:07.815464 2333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46315
I20260927 11:18:07.815527 2463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46315 every 8 connection(s)
I20260927 11:18:07.815828 2333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:18:07 dist-test-slave-lsvg krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:07.816658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 2333
I20260927 11:18:07.816756 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:07.816969 392 external_mini_cluster.cc:1428] Setting key 47b917260ec5cc7900f161f50dfa7100
I20260927 11:18:07.819590 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.809366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52657 (local address 127.0.98.62:42137)
0927 11:18:07.809421 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:07.809423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:07.809654 (+ 231us) server_negotiation.cc:407] Connection header received
0927 11:18:07.810176 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:07.810177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:07.810189 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:07.810241 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:07.811152 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.812006 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.813607 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.813834 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.813920 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.816535 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:07.816537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:07.816537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:07.816545 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:07.818176 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.818582 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.818583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.818584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.818647 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.818858 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.818859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.818859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.819008 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:07.819082 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:07.819436 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 11:18:07.819526 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 11:18:07.819837 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.809349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42137 (local address 127.0.98.2:52657)
0927 11:18:07.809541 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:07.809576 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:07.810077 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:07.810327 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:07.810330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:07.810741 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:07.811038 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:07.811045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.812107 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.812108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:07.813477 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:07.813479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.813930 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.813931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:07.814073 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.814676 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:07.814687 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:07.816430 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:07.818254 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:07.818260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:07.818262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:07.818510 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:07.818709 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:07.818709 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:07.818710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:07.818802 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:07.819086 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:07.819091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:07.819263 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 11:18:07.819392 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 11:18:07.819578 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":114,"threads_started":1}
I20260927 11:18:07.820334 2464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42137
I20260927 11:18:07.820451 2464 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:07.820637 2464 heartbeater.cc:511] Master 127.0.98.62:42137 requested a full tablet report, sending...
I20260927 11:18:07.821028 2120 ts_manager.cc:194] Registered new tserver with Master: 78c00bb8829b4553b75bcee4fcb25481 (127.0.98.2:46315)
I20260927 11:18:07.821487 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52657
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 11:18:07.828172 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.821676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40617 (local address 127.0.98.62:42137)
0927 11:18:07.821712 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:07.821714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:07.821744 (+ 30us) server_negotiation.cc:407] Connection header received
0927 11:18:07.821869 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:07.821869 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:07.821879 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:07.821945 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:07.822322 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.822767 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.823899 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.824085 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.824151 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.825259 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:07.825260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:07.825261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:07.825267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:07.826891 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.827304 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.827304 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.827304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.827352 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.827609 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.827610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.827610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.827754 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:07.827823 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:07.828025 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 11:18:07.828085 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:18:07.828197 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.821517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42137 (local address 127.0.98.2:40617)
0927 11:18:07.821677 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:07.821690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:07.821773 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:07.821950 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:07.821951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:07.822167 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:07.822237 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:07.822239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.822863 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.822864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:07.823793 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:07.823797 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.824166 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.824168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:07.824259 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.824703 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:07.824708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:07.825130 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:07.827058 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:07.827059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:07.827059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:07.827231 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:07.827426 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:07.827427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:07.827427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:07.827490 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:07.827848 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:07.827849 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:07.827891 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 11:18:07.828009 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 11:18:07.828121 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:18:07.844873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42137
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:07.925562 2473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:07.925779 2473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:07.925805 2473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:07.926935 2473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:07.926973 2473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:07.926994 2473 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 11:18:07.927006 2473 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:18:07.927023 2473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:07.928393 2473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:07.928582 2473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:07.928762 2473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:07.933907 2478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:07.934002 2479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:07.933940 2481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:07.934125 2473 server_base.cc:1034] running on GCE node
I20260927 11:18:07.934331 2473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:07.934571 2473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:07.935726 2473 hybrid_clock.cc:648] HybridClock initialized: now 1790507887935706 us; error 40 us; skew 500 ppm
Sep 27 11:18:07 dist-test-slave-lsvg krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:07.938225 2473 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:18:07.938647 2473 webserver.cc:460] Webserver started at http://127.0.98.3:43527/ using document root <none> and password file <none>
I20260927 11:18:07.938869 2473 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:07.938948 2473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:07.939116 2473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:07.940162 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "a35d08ffcc5148fd84df4c665d8bb841"
format_stamp: "Formatted at 2026-09-27 11:18:07 on dist-test-slave-lsvg"
server_key: "8f7df04243b1a53851d3e3a2f956670a"
server_key_iv: "f11b578e4b7cad576babd316653eb9b5"
server_key_version: "encryptionkey@0"
I20260927 11:18:07.940527 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a35d08ffcc5148fd84df4c665d8bb841"
format_stamp: "Formatted at 2026-09-27 11:18:07 on dist-test-slave-lsvg"
server_key: "8f7df04243b1a53851d3e3a2f956670a"
server_key_iv: "f11b578e4b7cad576babd316653eb9b5"
server_key_version: "encryptionkey@0"
I20260927 11:18:07.942037 2473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:07.942946 2488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:07.943141 2473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:07.943225 2473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "a35d08ffcc5148fd84df4c665d8bb841"
format_stamp: "Formatted at 2026-09-27 11:18:07 on dist-test-slave-lsvg"
server_key: "8f7df04243b1a53851d3e3a2f956670a"
server_key_iv: "f11b578e4b7cad576babd316653eb9b5"
server_key_version: "encryptionkey@0"
I20260927 11:18:07.943281 2473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:07.950923 2473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:07.953120 2473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:07.953315 2473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:07.953830 2473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:07.953877 2473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:07.953902 2473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:07.953946 2473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:07.960538 2473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38665
I20260927 11:18:07.960615 2603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38665 every 8 connection(s)
I20260927 11:18:07.960938 2473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 11:18:07 dist-test-slave-lsvg krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:07.964432 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.954339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58405 (local address 127.0.98.62:42137)
0927 11:18:07.954401 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:07.954403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:07.954624 (+ 221us) server_negotiation.cc:407] Connection header received
0927 11:18:07.955154 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:07.955155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:07.955166 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:07.955242 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:07.956162 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.957016 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.958782 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.959139 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.959224 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.961948 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:07.961950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:07.961950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:07.961957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:07.963198 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.963555 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.963555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.963556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.963609 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.963806 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.963807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.963807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.963969 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:07.964026 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:07.964295 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 11:18:07.964368 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:18:07.964589 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.954339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42137 (local address 127.0.98.3:58405)
0927 11:18:07.954510 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:07.954546 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:07.955042 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:07.955311 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:07.955315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:07.955713 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:07.956052 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:07.956058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.957124 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.957125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:07.958680 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:07.958683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.959230 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.959231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:07.959394 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.960044 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:07.960053 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:07.961818 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:07.963272 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:07.963277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:07.963279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:07.963490 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:07.963663 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:07.963664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:07.963664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:07.963755 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:07.964029 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:07.964033 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:07.964143 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 11:18:07.964230 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 11:18:07.964349 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260927 11:18:07.965121 2604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42137
I20260927 11:18:07.965220 2604 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:07.965402 2604 heartbeater.cc:511] Master 127.0.98.62:42137 requested a full tablet report, sending...
I20260927 11:18:07.965803 2121 ts_manager.cc:194] Registered new tserver with Master: a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665)
I20260927 11:18:07.966223 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58405
I20260927 11:18:07.969597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 2473
I20260927 11:18:07.969720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:07.969931 392 external_mini_cluster.cc:1428] Setting key a557da68699b8f127bf9c988d37c4d20
I20260927 11:18:07.970770 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:18:07.973362 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.966144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42137 (local address 127.0.98.3:55363)
0927 11:18:07.966315 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:07.966327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:07.966397 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:07.966540 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:07.966541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:07.966727 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:07.966786 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:07.966788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.967383 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.967384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:07.967875 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:07.967876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.968185 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.968185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:07.968235 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.968682 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:07.968687 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:07.968962 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:07.972388 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:07.972389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:07.972389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:07.972555 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:07.972782 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:07.972783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:07.972783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:07.972828 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:07.973111 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:07.973112 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:07.973156 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 11:18:07.973208 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 11:18:07.973288 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260927 11:18:07.973469 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.966224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55363 (local address 127.0.98.62:42137)
0927 11:18:07.966268 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:07.966270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:07.966369 (+ 99us) server_negotiation.cc:407] Connection header received
0927 11:18:07.966456 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:07.966456 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:07.966466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:07.966520 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:07.966855 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.967306 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.967948 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.968124 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.968178 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.969104 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:07.969106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:07.969106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:07.969113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:07.970719 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.972636 (+ 1917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.972637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.972637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.972710 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.972868 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.972868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.972868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.973009 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:07.973096 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:07.973255 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 11:18:07.973342 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 11:18:07 dist-test-slave-lsvg krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 11:18:07 dist-test-slave-lsvg krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:07.995003 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:07.987358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.0.98.62:42137)
0927 11:18:07.987421 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:07.987423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:07.987432 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:07.987459 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:07.987460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:07.987475 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:07.987537 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:07.988167 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.988944 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.989693 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:07.989898 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:07.990159 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:07.992455 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:07.992457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:07.992457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:07.992465 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:07.993887 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.994264 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.994266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.994266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.994322 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:07.994506 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:07.994507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:07.994507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:07.994618 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:07.994717 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:07.994817 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 11:18:07.994892 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:18:07.996481 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:07.998136 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:08.004530 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.000151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38665 (local address 127.0.0.1:56474)
0927 11:18:08.000315 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:08.000351 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:08.000416 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:08.000836 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:08.000840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:08.000843 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:08.000918 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.000920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.002325 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.002326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.003419 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.003421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.004213 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.004214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.004300 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.004315 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 11:18:08.004393 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 11:18:08.004447 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260927 11:18:08.004550 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.000490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.0.98.3:38665)
0927 11:18:08.000642 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.000648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.000659 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:18:08.000702 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.000704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.000721 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.000799 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:08.001031 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.002244 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.003504 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.004139 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.004194 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.004236 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.004417 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.004455 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260927 11:18:08.005259 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.000470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.0.98.2:46315)
0927 11:18:08.001662 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.001667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.001678 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:18:08.001713 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.001715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.001730 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.001806 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:08.001954 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.002767 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.003862 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.004436 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.004496 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.004552 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.005123 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.005175 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1136,"thread_start_us":74,"threads_started":1}
I20260927 11:18:08.005362 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.000270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.0.98.1:40711)
0927 11:18:08.000700 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.000704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.000712 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:08.000749 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.000751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.000765 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.001167 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:08.001230 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.002054 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.004265 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.004938 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.005013 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.005197 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.005223 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.005267 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":77,"threads_started":1}
I20260927 11:18:08.005453 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.000154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40711 (local address 127.0.0.1:51010)
0927 11:18:08.000447 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:08.000458 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:08.000508 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:08.001010 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:08.001010 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:08.001012 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:08.001088 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.001089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.002134 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.002135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.003419 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.003421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.005019 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.005020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.005096 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.005099 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:08.005343 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 11:18:08.005390 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":67,"threads_started":1}
I20260927 11:18:08.005359 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 9714a35af0234a5b96df062af9f82c8e (DEFAULT_TABLE table=test-table [id=3cddeecb31c8400d8a21d41ccd958b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:08.005703 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.000358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46315 (local address 127.0.0.1:48812)
0927 11:18:08.000857 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:08.000866 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:08.000926 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:08.001832 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:08.001832 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:08.001835 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:08.001870 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.001871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.002845 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.002846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.003779 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.003781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.004890 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.004891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.004955 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.004958 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:08.005028 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 11:18:08.005647 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":42,"threads_started":1}
I20260927 11:18:08.005744 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9714a35af0234a5b96df062af9f82c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:08.006186 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 9714a35af0234a5b96df062af9f82c8e (DEFAULT_TABLE table=test-table [id=3cddeecb31c8400d8a21d41ccd958b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:08.006510 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9714a35af0234a5b96df062af9f82c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:08.006487 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 9714a35af0234a5b96df062af9f82c8e (DEFAULT_TABLE table=test-table [id=3cddeecb31c8400d8a21d41ccd958b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:08.006788 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9714a35af0234a5b96df062af9f82c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:08.008579 2624 tablet_bootstrap.cc:492] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841: Bootstrap starting.
I20260927 11:18:08.009557 2624 tablet_bootstrap.cc:654] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:08.009856 2624 log.cc:824] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:08.010473 2627 tablet_bootstrap.cc:492] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481: Bootstrap starting.
I20260927 11:18:08.010517 2624 tablet_bootstrap.cc:492] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841: No bootstrap required, opened a new log
I20260927 11:18:08.010591 2624 ts_tablet_manager.cc:1397] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:18:08.010993 2627 tablet_bootstrap.cc:654] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:08.011296 2627 log.cc:824] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:08.011956 2627 tablet_bootstrap.cc:492] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481: No bootstrap required, opened a new log
I20260927 11:18:08.012008 2627 ts_tablet_manager.cc:1397] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:08.012292 2624 raft_consensus.cc:348] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } }
I20260927 11:18:08.012446 2624 raft_consensus.cc:374] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:08.012507 2624 raft_consensus.cc:729] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35d08ffcc5148fd84df4c665d8bb841, State: Initialized, Role: FOLLOWER
I20260927 11:18:08.012641 2624 consensus_queue.cc:260] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } }
I20260927 11:18:08.012899 2624 ts_tablet_manager.cc:1428] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:18:08.012940 2604 heartbeater.cc:503] Master 127.0.98.62:42137 was elected leader, sending a full tablet report...
I20260927 11:18:08.010224 2626 tablet_bootstrap.cc:492] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947: Bootstrap starting.
I20260927 11:18:08.013602 2626 tablet_bootstrap.cc:654] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:08.013731 2627 raft_consensus.cc:348] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } }
I20260927 11:18:08.013886 2627 raft_consensus.cc:374] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:08.013935 2627 raft_consensus.cc:729] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c00bb8829b4553b75bcee4fcb25481, State: Initialized, Role: FOLLOWER
I20260927 11:18:08.013936 2626 log.cc:824] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:08.014053 2627 consensus_queue.cc:260] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } }
I20260927 11:18:08.014319 2627 ts_tablet_manager.cc:1428] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:18:08.014611 2626 tablet_bootstrap.cc:492] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947: No bootstrap required, opened a new log
I20260927 11:18:08.014684 2626 ts_tablet_manager.cc:1397] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 11:18:08.016062 2626 raft_consensus.cc:348] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } }
I20260927 11:18:08.016194 2626 raft_consensus.cc:374] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:08.016237 2626 raft_consensus.cc:729] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f08c903b69472c872448f912869947, State: Initialized, Role: FOLLOWER
I20260927 11:18:08.016360 2626 consensus_queue.cc:260] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } }
I20260927 11:18:08.016593 2626 ts_tablet_manager.cc:1428] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:18:08.016669 2324 heartbeater.cc:503] Master 127.0.98.62:42137 was elected leader, sending a full tablet report...
I20260927 11:18:08.016726 2464 heartbeater.cc:503] Master 127.0.98.62:42137 was elected leader, sending a full tablet report...
W20260927 11:18:08.066439 2465 tablet.cc:2369] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:08.119896 2632 raft_consensus.cc:482] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:08.120028 2632 raft_consensus.cc:504] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } }
I20260927 11:18:08.120434 2632 leader_election.cc:288] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665), 78c00bb8829b4553b75bcee4fcb25481 (127.0.98.2:46315)
I20260927 11:18:08.123510 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.120752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48249 (local address 127.0.98.3:38665)
0927 11:18:08.120814 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.120816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.120827 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:18:08.120857 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.120858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.120870 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.120979 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:08.121142 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.121735 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.122803 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.123169 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.123223 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.123231 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.123394 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.123440 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 11:18:08.123528 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.120672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38665 (local address 127.0.98.1:48249)
0927 11:18:08.120726 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:08.120739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:08.120784 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:08.120995 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:08.120996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:08.120998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:08.121076 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.121078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.121811 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.121812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.122710 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.122712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.123245 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.123246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.123320 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.123323 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:08.123371 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 11:18:08.123408 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":46}
I20260927 11:18:08.123553 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.120918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55545 (local address 127.0.98.2:46315)
0927 11:18:08.120983 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.120985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.121158 (+ 173us) server_negotiation.cc:407] Connection header received
0927 11:18:08.121231 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.121232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.121241 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.121312 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:08.121446 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.121971 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.122843 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.123183 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.123270 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.123280 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.123451 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.123487 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 11:18:08.123656 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.120792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46315 (local address 127.0.98.1:55545)
0927 11:18:08.121099 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:08.121110 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:08.121171 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:08.121283 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:08.121284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:08.121285 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:08.121351 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.121352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.122050 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.122050 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.122740 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.122742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.123288 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.123289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.123352 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.123371 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 11:18:08.123435 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 11:18:08.123472 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":141,"threads_started":1}
I20260927 11:18:08.123965 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9714a35af0234a5b96df062af9f82c8e" candidate_uuid: "c7f08c903b69472c872448f912869947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c00bb8829b4553b75bcee4fcb25481" is_pre_election: true
I20260927 11:18:08.123974 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9714a35af0234a5b96df062af9f82c8e" candidate_uuid: "c7f08c903b69472c872448f912869947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35d08ffcc5148fd84df4c665d8bb841" is_pre_election: true
I20260927 11:18:08.124118 2418 raft_consensus.cc:2454] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7f08c903b69472c872448f912869947 in term 0.
I20260927 11:18:08.124130 2558 raft_consensus.cc:2454] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7f08c903b69472c872448f912869947 in term 0.
I20260927 11:18:08.124332 2212 leader_election.cc:302] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a35d08ffcc5148fd84df4c665d8bb841, c7f08c903b69472c872448f912869947; no voters:
I20260927 11:18:08.124456 2632 raft_consensus.cc:2781] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:08.124536 2632 raft_consensus.cc:482] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:08.124584 2632 raft_consensus.cc:3037] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:08.125339 2632 raft_consensus.cc:504] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } }
I20260927 11:18:08.125481 2632 leader_election.cc:288] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [CANDIDATE]: Term 1 election: Requested vote from peers a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665), 78c00bb8829b4553b75bcee4fcb25481 (127.0.98.2:46315)
I20260927 11:18:08.125648 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9714a35af0234a5b96df062af9f82c8e" candidate_uuid: "c7f08c903b69472c872448f912869947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35d08ffcc5148fd84df4c665d8bb841"
I20260927 11:18:08.125674 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9714a35af0234a5b96df062af9f82c8e" candidate_uuid: "c7f08c903b69472c872448f912869947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78c00bb8829b4553b75bcee4fcb25481"
I20260927 11:18:08.125721 2558 raft_consensus.cc:3037] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:08.125756 2418 raft_consensus.cc:3037] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:08.126437 2558 raft_consensus.cc:2454] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7f08c903b69472c872448f912869947 in term 1.
I20260927 11:18:08.126467 2418 raft_consensus.cc:2454] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7f08c903b69472c872448f912869947 in term 1.
I20260927 11:18:08.126603 2212 leader_election.cc:302] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c00bb8829b4553b75bcee4fcb25481, c7f08c903b69472c872448f912869947; no voters:
I20260927 11:18:08.126725 2632 raft_consensus.cc:2781] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:08.126896 2632 raft_consensus.cc:686] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [term 1 LEADER]: Becoming Leader. State: Replica: c7f08c903b69472c872448f912869947, State: Running, Role: LEADER
I20260927 11:18:08.126998 2632 consensus_queue.cc:237] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } }
I20260927 11:18:08.127836 2121 catalog_manager.cc:5495] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7f08c903b69472c872448f912869947 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c7f08c903b69472c872448f912869947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } health_report { overall_health: UNKNOWN } } }
W20260927 11:18:08.157637 2325 tablet.cc:2369] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 11:18:08 dist-test-slave-lsvg krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:08.176841 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.169884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.0.98.62:42137)
0927 11:18:08.169992 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.169994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.170129 (+ 135us) server_negotiation.cc:407] Connection header received
0927 11:18:08.170214 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.170214 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.170224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.170290 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:08.170784 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.171262 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.171993 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.172174 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.172252 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.174372 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:08.174373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:08.174374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:08.174387 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:08.175709 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:08.176093 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:08.176095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:08.176095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:08.176154 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:08.176332 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:08.176333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:08.176333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:08.176441 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:08.176511 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.176653 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.176731 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 11:18:08.176913 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.169851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42137 (local address 127.0.0.1:37228)
0927 11:18:08.170050 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:08.170061 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:08.170150 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:08.170295 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:08.170296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:08.170644 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:08.170691 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.170694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.171349 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.171350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.171899 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.171902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.172252 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.172253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.172303 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.172855 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:08.172860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:08.174288 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:08.175800 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:08.175807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:08.175810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:08.176011 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:08.176215 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:08.176216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:08.176217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:08.176273 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:08.176502 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:08.176509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:08.176552 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 11:18:08.176632 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 11:18:08.176786 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":58,"thread_start_us":86,"threads_started":1}
I20260927 11:18:08.179086 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37228:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 11:18:08.179286 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:08.180737 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2d08ae7b744e5993c0ca9cf7d4eda8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=111c3f2ecf8a400e9273b4014559fd5b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:18:08.180769 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2d08ae7b744e5993c0ca9cf7d4eda8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=111c3f2ecf8a400e9273b4014559fd5b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:18:08.180822 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2d08ae7b744e5993c0ca9cf7d4eda8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=111c3f2ecf8a400e9273b4014559fd5b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:18:08.180926 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2d08ae7b744e5993c0ca9cf7d4eda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:08.180926 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2d08ae7b744e5993c0ca9cf7d4eda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:08.180966 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2d08ae7b744e5993c0ca9cf7d4eda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:08.182371 2627 tablet_bootstrap.cc:492] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481: Bootstrap starting.
I20260927 11:18:08.182451 2626 tablet_bootstrap.cc:492] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947: Bootstrap starting.
I20260927 11:18:08.182454 2624 tablet_bootstrap.cc:492] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841: Bootstrap starting.
I20260927 11:18:08.182857 2626 tablet_bootstrap.cc:654] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:08.182961 2627 tablet_bootstrap.cc:654] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:08.182987 2624 tablet_bootstrap.cc:654] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:08.183641 2627 tablet_bootstrap.cc:492] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481: No bootstrap required, opened a new log
I20260927 11:18:08.183720 2627 ts_tablet_manager.cc:1397] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:08.183861 2624 tablet_bootstrap.cc:492] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841: No bootstrap required, opened a new log
I20260927 11:18:08.183928 2624 ts_tablet_manager.cc:1397] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:08.183903 2627 raft_consensus.cc:348] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.183987 2627 raft_consensus.cc:374] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:08.184008 2627 raft_consensus.cc:729] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c00bb8829b4553b75bcee4fcb25481, State: Initialized, Role: FOLLOWER
I20260927 11:18:08.184047 2626 tablet_bootstrap.cc:492] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947: No bootstrap required, opened a new log
I20260927 11:18:08.184054 2627 consensus_queue.cc:260] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.184053 2624 raft_consensus.cc:348] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.184087 2626 ts_tablet_manager.cc:1397] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:08.184109 2624 raft_consensus.cc:374] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:08.184146 2624 raft_consensus.cc:729] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35d08ffcc5148fd84df4c665d8bb841, State: Initialized, Role: FOLLOWER
I20260927 11:18:08.184175 2627 ts_tablet_manager.cc:1428] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:08.184187 2624 consensus_queue.cc:260] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.184271 2624 ts_tablet_manager.cc:1428] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:08.184223 2626 raft_consensus.cc:348] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.184249 2631 tablet_replica.cc:439] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } }
I20260927 11:18:08.184315 2626 raft_consensus.cc:374] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:08.184342 2626 raft_consensus.cc:729] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f08c903b69472c872448f912869947, State: Initialized, Role: FOLLOWER
I20260927 11:18:08.184345 2631 tablet_replica.cc:442] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:08.184410 2626 consensus_queue.cc:260] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.184515 2626 ts_tablet_manager.cc:1428] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:08.184536 2629 tablet_replica.cc:439] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } }
I20260927 11:18:08.184610 2629 tablet_replica.cc:442] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:08.184913 2632 tablet_replica.cc:439] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } }
I20260927 11:18:08.184998 2632 tablet_replica.cc:442] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:08.188488 2631 raft_consensus.cc:482] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:08.188556 2631 raft_consensus.cc:504] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.188916 2631 leader_election.cc:288] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7f08c903b69472c872448f912869947 (127.0.98.1:40711), a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665)
I20260927 11:18:08.191658 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.189100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40711 (local address 127.0.98.2:47785)
0927 11:18:08.189147 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:08.189159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:08.189202 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:08.189343 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:08.189344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:08.189346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:08.189462 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.189464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.190128 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.190129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.190856 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.190858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.191383 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.191384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.191468 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.191470 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:18:08.191539 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 11:18:08.191582 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 11:18:08.191648 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.189140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47785 (local address 127.0.98.1:40711)
0927 11:18:08.189223 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.189224 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.189232 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:08.189256 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.189257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.189267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.189335 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:08.189531 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.190045 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.190943 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.191299 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.191379 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.191389 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.191547 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.191582 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 11:18:08.192036 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d08ae7b744e5993c0ca9cf7d4eda8" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7f08c903b69472c872448f912869947" is_pre_election: true
I20260927 11:18:08.192159 2278 raft_consensus.cc:2454] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 0.
I20260927 11:18:08.192381 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.189105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38665 (local address 127.0.98.2:59471)
0927 11:18:08.189463 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:08.189488 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:08.189562 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:08.189736 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:08.189737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:08.189739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:08.189809 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.189816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.190422 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.190423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.191155 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.191157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.191667 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.191668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.192143 (+ 475us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.192148 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:18:08.192241 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 11:18:08.192300 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":150,"thread_start_us":93,"threads_started":1}
I20260927 11:18:08.192361 2350 leader_election.cc:302] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c00bb8829b4553b75bcee4fcb25481, c7f08c903b69472c872448f912869947; no voters:
I20260927 11:18:08.192427 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.189133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59471 (local address 127.0.98.3:38665)
0927 11:18:08.189188 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.189191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.189536 (+ 345us) server_negotiation.cc:407] Connection header received
0927 11:18:08.189645 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.189646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.189655 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.189732 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:08.189886 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.190340 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.191240 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.191593 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.191668 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.191676 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.192331 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.192380 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:18:08.192528 2631 raft_consensus.cc:2781] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:08.192559 2631 raft_consensus.cc:482] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:08.192577 2631 raft_consensus.cc:3037] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:08.192647 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d08ae7b744e5993c0ca9cf7d4eda8" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35d08ffcc5148fd84df4c665d8bb841" is_pre_election: true
I20260927 11:18:08.192723 2558 raft_consensus.cc:2454] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 0.
I20260927 11:18:08.193456 2631 raft_consensus.cc:504] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.193593 2631 leader_election.cc:288] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 election: Requested vote from peers c7f08c903b69472c872448f912869947 (127.0.98.1:40711), a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665)
I20260927 11:18:08.193730 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d08ae7b744e5993c0ca9cf7d4eda8" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7f08c903b69472c872448f912869947"
I20260927 11:18:08.193773 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d08ae7b744e5993c0ca9cf7d4eda8" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35d08ffcc5148fd84df4c665d8bb841"
I20260927 11:18:08.193810 2278 raft_consensus.cc:3037] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:08.193841 2558 raft_consensus.cc:3037] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:08.194561 2558 raft_consensus.cc:2454] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 1.
I20260927 11:18:08.194561 2278 raft_consensus.cc:2454] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 1.
I20260927 11:18:08.194730 2352 leader_election.cc:302] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c00bb8829b4553b75bcee4fcb25481, a35d08ffcc5148fd84df4c665d8bb841; no voters:
I20260927 11:18:08.194833 2631 raft_consensus.cc:2781] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:08.194980 2631 raft_consensus.cc:686] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [term 1 LEADER]: Becoming Leader. State: Replica: 78c00bb8829b4553b75bcee4fcb25481, State: Running, Role: LEADER
I20260927 11:18:08.195091 2631 consensus_queue.cc:237] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.195516 2646 tablet_replica.cc:439] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481: TxnStatusTablet state changed. Reason: New leader 78c00bb8829b4553b75bcee4fcb25481. Latest consensus state: current_term: 1 leader_uuid: "78c00bb8829b4553b75bcee4fcb25481" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } }
I20260927 11:18:08.195609 2646 tablet_replica.cc:442] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:18:08.195779 2648 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 11:18:08.195910 2648 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 11:18:08.195858 2121 catalog_manager.cc:5495] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c00bb8829b4553b75bcee4fcb25481 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "78c00bb8829b4553b75bcee4fcb25481" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } health_report { overall_health: UNKNOWN } } }
I20260927 11:18:08.201880 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.199090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.0.98.2:46315)
0927 11:18:08.199128 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.199130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.199205 (+ 75us) server_negotiation.cc:407] Connection header received
0927 11:18:08.199297 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.199298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.199311 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.199365 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:08.199503 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.199950 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.201120 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.201329 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.201411 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.201598 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:08.201684 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:08.201740 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.201812 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.201828 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:18:08.201952 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.199061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46315 (local address 127.0.0.1:48826)
0927 11:18:08.199145 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:08.199159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:08.199223 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:08.199389 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:08.199390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:08.199392 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:18:08.199439 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.199441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.200018 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.200019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.200993 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.200996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.201422 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.201423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.201492 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.201512 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:18:08.201744 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:18:08.201754 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 11:18:08.201811 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 11:18:08.201862 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 11:18:08.203701 2558 raft_consensus.cc:1261] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Refusing update from remote peer 78c00bb8829b4553b75bcee4fcb25481: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:08.203758 2278 raft_consensus.cc:1261] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Refusing update from remote peer 78c00bb8829b4553b75bcee4fcb25481: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:08.204010 2646 consensus_queue.cc:1035] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:08.204144 2631 consensus_queue.cc:1035] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:08.205267 2632 tablet_replica.cc:439] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947: TxnStatusTablet state changed. Reason: New leader 78c00bb8829b4553b75bcee4fcb25481. Latest consensus state: current_term: 1 leader_uuid: "78c00bb8829b4553b75bcee4fcb25481" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } }
I20260927 11:18:08.205358 2632 tablet_replica.cc:442] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:08.205961 2629 tablet_replica.cc:439] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841: TxnStatusTablet state changed. Reason: New leader 78c00bb8829b4553b75bcee4fcb25481. Latest consensus state: current_term: 1 leader_uuid: "78c00bb8829b4553b75bcee4fcb25481" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } }
I20260927 11:18:08.206054 2629 tablet_replica.cc:442] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:08.206854 2631 tablet_replica.cc:439] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78c00bb8829b4553b75bcee4fcb25481" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } }
I20260927 11:18:08.206943 2631 tablet_replica.cc:442] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:18:08.207028 2629 tablet_replica.cc:439] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78c00bb8829b4553b75bcee4fcb25481" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } }
I20260927 11:18:08.207101 2629 tablet_replica.cc:442] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P a35d08ffcc5148fd84df4c665d8bb841: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:08.207340 2652 mvcc.cc:205] Tried to move back new op lower bound from 7333920310080217088 to 7333920310047834112. Current Snapshot: MvccSnapshot[applied={T|T < 7333920310080217088}]
W20260927 11:18:08.211628 2605 tablet.cc:2369] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:08.212713 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.209469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.0.98.1:40711)
0927 11:18:08.209514 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.209516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.209526 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:08.209565 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.209566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.209585 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.209653 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:08.209827 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.210516 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.211889 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.212079 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.212176 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.212355 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:08.212464 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:08.212559 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.212582 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.212608 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:18:08.215260 2646 tablet_replica.cc:439] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78c00bb8829b4553b75bcee4fcb25481" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } }
I20260927 11:18:08.215368 2646 tablet_replica.cc:442] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P 78c00bb8829b4553b75bcee4fcb25481: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:18:08.216501 2632 tablet_replica.cc:439] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78c00bb8829b4553b75bcee4fcb25481" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } } }
I20260927 11:18:08.216595 2632 tablet_replica.cc:442] T 4e2d08ae7b744e5993c0ca9cf7d4eda8 P c7f08c903b69472c872448f912869947: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:08.220880 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.218355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48021 (local address 127.0.98.2:46315)
0927 11:18:08.218443 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.218445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.218574 (+ 129us) server_negotiation.cc:407] Connection header received
0927 11:18:08.218640 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.218641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.218652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.218706 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:08.218842 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.219305 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.220133 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.220340 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.220427 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.220611 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:08.220644 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:08.220692 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.220755 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.220775 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 11:18:08.220883 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.218299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46315 (local address 127.0.98.1:48021)
0927 11:18:08.218510 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:08.218520 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:08.218589 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:08.218708 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:08.218709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:08.218711 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:18:08.218774 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.218776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.219394 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.219395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.220057 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.220059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.220428 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.220429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.220496 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.220507 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:18:08.220698 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:18:08.220700 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:18:08.220751 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 11:18:08.220790 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260927 11:18:08.223316 2558 raft_consensus.cc:1261] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Refusing update from remote peer c7f08c903b69472c872448f912869947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:08.223431 2418 raft_consensus.cc:1261] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Refusing update from remote peer c7f08c903b69472c872448f912869947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:08.223575 2634 consensus_queue.cc:1035] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:08.223680 2632 consensus_queue.cc:1035] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:08.225466 2651 mvcc.cc:205] Tried to move back new op lower bound from 7333920310160908288 to 7333920309769031680. Current Snapshot: MvccSnapshot[applied={T|T < 7333920310160908288}]
I20260927 11:18:08.232555 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.229820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38393 (local address 127.0.98.1:40711)
0927 11:18:08.229863 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.229865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.229919 (+ 54us) server_negotiation.cc:407] Connection header received
0927 11:18:08.229988 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.229989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.229997 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.230038 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:08.230182 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.230724 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.231820 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.232083 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.232144 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.232339 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:08.232366 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:08.232434 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.232479 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.232498 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:18:08.232594 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.229786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40711 (local address 127.0.98.2:38393)
0927 11:18:08.229860 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:08.229879 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:08.229931 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:08.230041 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:08.230042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:08.230043 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:18:08.230103 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.230104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.230801 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.230802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.231744 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.231746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.232179 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.232180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.232249 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.232265 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:18:08.232432 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:18:08.232434 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:18:08.232478 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 11:18:08.232518 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 11:18:08.239588 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:08.239692 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:08.240443 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78: Sending DeleteTablet for 3 replicas of tablet 9714a35af0234a5b96df062af9f82c8e
I20260927 11:18:08.240803 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 9714a35af0234a5b96df062af9f82c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48812
I20260927 11:18:08.240800 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 9714a35af0234a5b96df062af9f82c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56474
I20260927 11:18:08.241176 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 9714a35af0234a5b96df062af9f82c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51010
I20260927 11:18:08.241425 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:08.241566 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:08.242741 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 393d384c48d74effaac46c840aa8dcb5 (DEFAULT_TABLE table=test-table [id=242a64ed21c04d9884e8d7d394a9986f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:08.242846 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 393d384c48d74effaac46c840aa8dcb5 (DEFAULT_TABLE table=test-table [id=242a64ed21c04d9884e8d7d394a9986f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:08.242863 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393d384c48d74effaac46c840aa8dcb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:08.242962 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393d384c48d74effaac46c840aa8dcb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:08.244242 2626 tablet_bootstrap.cc:492] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947: Bootstrap starting.
I20260927 11:18:08.244283 2624 tablet_bootstrap.cc:492] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841: Bootstrap starting.
I20260927 11:18:08.244666 2624 tablet_bootstrap.cc:654] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:08.244666 2626 tablet_bootstrap.cc:654] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:08.245314 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 393d384c48d74effaac46c840aa8dcb5 (DEFAULT_TABLE table=test-table [id=242a64ed21c04d9884e8d7d394a9986f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:08.245458 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393d384c48d74effaac46c840aa8dcb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:08.246822 2627 tablet_bootstrap.cc:492] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481: Bootstrap starting.
I20260927 11:18:08.247251 2627 tablet_bootstrap.cc:654] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:08.241858 2671 tablet_replica.cc:330] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841: stopping tablet replica
I20260927 11:18:08.247376 2671 raft_consensus.cc:2229] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:08.247448 2671 raft_consensus.cc:2258] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:08.247902 2671 ts_tablet_manager.cc:1905] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:08.249179 2672 tablet_replica.cc:330] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947: stopping tablet replica
I20260927 11:18:08.249336 2670 tablet_replica.cc:330] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481: stopping tablet replica
I20260927 11:18:08.249483 2670 raft_consensus.cc:2229] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:08.249434 2626 tablet_bootstrap.cc:492] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947: No bootstrap required, opened a new log
I20260927 11:18:08.249596 2626 ts_tablet_manager.cc:1397] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 11:18:08.249622 2672 raft_consensus.cc:2229] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:08.249720 2672 raft_consensus.cc:2258] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:08.249782 2671 ts_tablet_manager.cc:1918] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:08.249760 2626 raft_consensus.cc:348] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.249830 2626 raft_consensus.cc:374] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:08.249827 2670 raft_consensus.cc:2258] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:08.249857 2626 raft_consensus.cc:729] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f08c903b69472c872448f912869947, State: Initialized, Role: FOLLOWER
I20260927 11:18:08.249838 2671 log.cc:1196] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/9714a35af0234a5b96df062af9f82c8e
I20260927 11:18:08.249903 2626 consensus_queue.cc:260] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.250015 2626 ts_tablet_manager.cc:1428] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:08.250133 2671 ts_tablet_manager.cc:1939] T 9714a35af0234a5b96df062af9f82c8e P a35d08ffcc5148fd84df4c665d8bb841: Deleting consensus metadata
I20260927 11:18:08.250506 2108 catalog_manager.cc:4841] TS a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665): tablet 9714a35af0234a5b96df062af9f82c8e (table test-table [id=3cddeecb31c8400d8a21d41ccd958b07]) successfully deleted
I20260927 11:18:08.250821 2670 ts_tablet_manager.cc:1905] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:08.251134 2624 tablet_bootstrap.cc:492] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841: No bootstrap required, opened a new log
I20260927 11:18:08.251179 2624 ts_tablet_manager.cc:1397] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260927 11:18:08.251302 2624 raft_consensus.cc:348] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.251359 2624 raft_consensus.cc:374] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:08.251379 2624 raft_consensus.cc:729] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35d08ffcc5148fd84df4c665d8bb841, State: Initialized, Role: FOLLOWER
I20260927 11:18:08.251415 2624 consensus_queue.cc:260] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.251487 2624 ts_tablet_manager.cc:1428] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:08.252383 2670 ts_tablet_manager.cc:1918] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:08.252456 2670 log.cc:1196] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/9714a35af0234a5b96df062af9f82c8e
I20260927 11:18:08.252694 2670 ts_tablet_manager.cc:1939] T 9714a35af0234a5b96df062af9f82c8e P 78c00bb8829b4553b75bcee4fcb25481: Deleting consensus metadata
I20260927 11:18:08.253111 2108 catalog_manager.cc:4841] TS 78c00bb8829b4553b75bcee4fcb25481 (127.0.98.2:46315): tablet 9714a35af0234a5b96df062af9f82c8e (table test-table [id=3cddeecb31c8400d8a21d41ccd958b07]) successfully deleted
I20260927 11:18:08.253305 2627 tablet_bootstrap.cc:492] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481: No bootstrap required, opened a new log
I20260927 11:18:08.253357 2627 ts_tablet_manager.cc:1397] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260927 11:18:08.253482 2627 raft_consensus.cc:348] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.253538 2627 raft_consensus.cc:374] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:08.253557 2627 raft_consensus.cc:729] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c00bb8829b4553b75bcee4fcb25481, State: Initialized, Role: FOLLOWER
I20260927 11:18:08.253600 2627 consensus_queue.cc:260] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.253679 2627 ts_tablet_manager.cc:1428] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:08.254108 2672 ts_tablet_manager.cc:1905] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:08.255460 2672 ts_tablet_manager.cc:1918] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:08.255586 2672 log.cc:1196] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/wal/wals/9714a35af0234a5b96df062af9f82c8e
I20260927 11:18:08.255896 2672 ts_tablet_manager.cc:1939] T 9714a35af0234a5b96df062af9f82c8e P c7f08c903b69472c872448f912869947: Deleting consensus metadata
I20260927 11:18:08.256254 2106 catalog_manager.cc:4841] TS c7f08c903b69472c872448f912869947 (127.0.98.1:40711): tablet 9714a35af0234a5b96df062af9f82c8e (table test-table [id=3cddeecb31c8400d8a21d41ccd958b07]) successfully deleted
I20260927 11:18:08.339084 2646 raft_consensus.cc:482] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:08.339193 2646 raft_consensus.cc:504] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.339363 2646 leader_election.cc:288] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7f08c903b69472c872448f912869947 (127.0.98.1:40711), a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665)
I20260927 11:18:08.339583 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393d384c48d74effaac46c840aa8dcb5" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35d08ffcc5148fd84df4c665d8bb841" is_pre_election: true
I20260927 11:18:08.339684 2558 raft_consensus.cc:2454] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 0.
I20260927 11:18:08.339669 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393d384c48d74effaac46c840aa8dcb5" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7f08c903b69472c872448f912869947" is_pre_election: true
I20260927 11:18:08.339768 2278 raft_consensus.cc:2454] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 0.
I20260927 11:18:08.339972 2352 leader_election.cc:302] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c00bb8829b4553b75bcee4fcb25481, a35d08ffcc5148fd84df4c665d8bb841; no voters:
I20260927 11:18:08.340058 2646 raft_consensus.cc:2781] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:08.340103 2646 raft_consensus.cc:482] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:08.340123 2646 raft_consensus.cc:3037] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:08.340982 2646 raft_consensus.cc:504] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.341140 2646 leader_election.cc:288] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 election: Requested vote from peers c7f08c903b69472c872448f912869947 (127.0.98.1:40711), a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665)
I20260927 11:18:08.341274 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393d384c48d74effaac46c840aa8dcb5" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7f08c903b69472c872448f912869947"
I20260927 11:18:08.341342 2278 raft_consensus.cc:3037] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:08.341349 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393d384c48d74effaac46c840aa8dcb5" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35d08ffcc5148fd84df4c665d8bb841"
I20260927 11:18:08.341423 2558 raft_consensus.cc:3037] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:08.342126 2278 raft_consensus.cc:2454] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 1.
I20260927 11:18:08.342186 2558 raft_consensus.cc:2454] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 1.
I20260927 11:18:08.342317 2350 leader_election.cc:302] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c00bb8829b4553b75bcee4fcb25481, c7f08c903b69472c872448f912869947; no voters:
I20260927 11:18:08.342451 2646 raft_consensus.cc:2781] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:08.342489 2646 raft_consensus.cc:686] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [term 1 LEADER]: Becoming Leader. State: Replica: 78c00bb8829b4553b75bcee4fcb25481, State: Running, Role: LEADER
I20260927 11:18:08.342528 2646 consensus_queue.cc:237] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:08.343161 2120 catalog_manager.cc:5495] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c00bb8829b4553b75bcee4fcb25481 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "78c00bb8829b4553b75bcee4fcb25481" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } health_report { overall_health: UNKNOWN } } }
I20260927 11:18:08.364929 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.362936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.0.98.2:46315)
0927 11:18:08.362995 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.362997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.363005 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:08.363030 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.363031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.363040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.363079 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:08.363167 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.363646 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.364329 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.364578 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.364637 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.364749 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:08.364775 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:08.364812 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.364850 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.364870 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:18:08.369395 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.367280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33179 (local address 127.0.98.2:46315)
0927 11:18:08.367331 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:08.367333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:08.367342 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:08.367423 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:08.367424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:08.367433 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:08.367483 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:08.367590 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.368041 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.368740 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.368928 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.368996 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.369156 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:08.369185 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:08.369239 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:08.369291 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 11:18:08.369313 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":34}
I20260927 11:18:08.369402 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:08.367195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46315 (local address 127.0.98.2:33179)
0927 11:18:08.367251 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:08.367266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:08.367319 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:08.367484 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:08.367484 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:08.367485 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:18:08.367525 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.367526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.368118 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.368119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.368658 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:08.368659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:08.368993 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:08.368994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:08.369076 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:08.369080 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:18:08.369240 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:18:08.369242 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:18:08.369289 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 11:18:08.369347 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":3}
I20260927 11:18:08.371227 2278 raft_consensus.cc:1261] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Refusing update from remote peer 78c00bb8829b4553b75bcee4fcb25481: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:08.371258 2558 raft_consensus.cc:1261] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Refusing update from remote peer 78c00bb8829b4553b75bcee4fcb25481: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:08.371447 2659 consensus_queue.cc:1035] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:08.371546 2631 consensus_queue.cc:1035] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:08.372774 2653 mvcc.cc:205] Tried to move back new op lower bound from 7333920310767362048 to 7333920310651334656. Current Snapshot: MvccSnapshot[applied={T|T < 7333920310767362048}]
W20260927 11:18:08.376253 2651 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 11:18:09.894809 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:09.894917 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:09.895718 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78: Sending DeleteTablet for 3 replicas of tablet 393d384c48d74effaac46c840aa8dcb5
I20260927 11:18:09.895998 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 393d384c48d74effaac46c840aa8dcb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51010
I20260927 11:18:09.896073 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 393d384c48d74effaac46c840aa8dcb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48812
I20260927 11:18:09.896166 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 393d384c48d74effaac46c840aa8dcb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56474
I20260927 11:18:09.896579 2701 tablet_replica.cc:330] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841: stopping tablet replica
I20260927 11:18:09.896636 2701 raft_consensus.cc:2229] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:09.896667 2701 raft_consensus.cc:2258] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:09.896788 2699 tablet_replica.cc:330] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947: stopping tablet replica
I20260927 11:18:09.896859 2699 raft_consensus.cc:2229] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:09.896934 2699 raft_consensus.cc:2258] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:09.896983 2701 ts_tablet_manager.cc:1905] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:09.897234 2699 ts_tablet_manager.cc:1905] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:09.898985 2700 tablet_replica.cc:330] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481: stopping tablet replica
I20260927 11:18:09.899058 2700 raft_consensus.cc:2229] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:09.899115 2699 ts_tablet_manager.cc:1918] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 11:18:09.899171 2700 raft_consensus.cc:2258] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:09.899185 2699 log.cc:1196] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/wal/wals/393d384c48d74effaac46c840aa8dcb5
I20260927 11:18:09.899292 2701 ts_tablet_manager.cc:1918] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 11:18:09.899402 2701 log.cc:1196] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/393d384c48d74effaac46c840aa8dcb5
I20260927 11:18:09.899421 2699 ts_tablet_manager.cc:1939] T 393d384c48d74effaac46c840aa8dcb5 P c7f08c903b69472c872448f912869947: Deleting consensus metadata
I20260927 11:18:09.899430 2700 ts_tablet_manager.cc:1905] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:09.899732 2701 ts_tablet_manager.cc:1939] T 393d384c48d74effaac46c840aa8dcb5 P a35d08ffcc5148fd84df4c665d8bb841: Deleting consensus metadata
I20260927 11:18:09.899729 2106 catalog_manager.cc:4841] TS c7f08c903b69472c872448f912869947 (127.0.98.1:40711): tablet 393d384c48d74effaac46c840aa8dcb5 (table test-table [id=242a64ed21c04d9884e8d7d394a9986f]) successfully deleted
I20260927 11:18:09.900022 2108 catalog_manager.cc:4841] TS a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665): tablet 393d384c48d74effaac46c840aa8dcb5 (table test-table [id=242a64ed21c04d9884e8d7d394a9986f]) successfully deleted
I20260927 11:18:09.900962 2700 ts_tablet_manager.cc:1918] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 11:18:09.901008 2700 log.cc:1196] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/393d384c48d74effaac46c840aa8dcb5
I20260927 11:18:09.901264 2700 ts_tablet_manager.cc:1939] T 393d384c48d74effaac46c840aa8dcb5 P 78c00bb8829b4553b75bcee4fcb25481: Deleting consensus metadata
I20260927 11:18:09.901587 2108 catalog_manager.cc:4841] TS 78c00bb8829b4553b75bcee4fcb25481 (127.0.98.2:46315): tablet 393d384c48d74effaac46c840aa8dcb5 (table test-table [id=242a64ed21c04d9884e8d7d394a9986f]) successfully deleted
Sep 27 11:18:09 dist-test-slave-lsvg krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507889, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 11:18:09 dist-test-slave-lsvg krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507889, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:09.921175 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:09.913439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.0.98.62:42137)
0927 11:18:09.913578 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:09.913580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:09.913592 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:18:09.913652 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:09.913652 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:09.913663 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:09.913731 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:09.914267 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:09.914783 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:09.915640 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:09.915838 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:09.915902 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:09.918494 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:09.918496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:09.918496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:09.918503 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:09.920095 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:09.920449 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:09.920451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:09.920452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:09.920524 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:09.920691 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:09.920692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:09.920692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:09.920804 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:09.920880 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:09.920976 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 11:18:09.921040 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260927 11:18:09.922750 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:09.922894 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:09.924191 2258 tablet_service.cc:1467] Processing CreateTablet for tablet e286e0321de64f33b037838bbed82a12 (DEFAULT_TABLE table=test-table [id=97e8c21aa9534659913201ba2e1d52df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:09.924223 2538 tablet_service.cc:1467] Processing CreateTablet for tablet e286e0321de64f33b037838bbed82a12 (DEFAULT_TABLE table=test-table [id=97e8c21aa9534659913201ba2e1d52df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:09.924338 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e286e0321de64f33b037838bbed82a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:09.924340 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e286e0321de64f33b037838bbed82a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:09.925156 2398 tablet_service.cc:1467] Processing CreateTablet for tablet e286e0321de64f33b037838bbed82a12 (DEFAULT_TABLE table=test-table [id=97e8c21aa9534659913201ba2e1d52df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:09.925282 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e286e0321de64f33b037838bbed82a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:09.925909 2710 tablet_bootstrap.cc:492] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947: Bootstrap starting.
I20260927 11:18:09.926625 2711 tablet_bootstrap.cc:492] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481: Bootstrap starting.
I20260927 11:18:09.926767 2710 tablet_bootstrap.cc:654] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:09.926992 2711 tablet_bootstrap.cc:654] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:09.927449 2714 tablet_bootstrap.cc:492] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841: Bootstrap starting.
I20260927 11:18:09.927603 2711 tablet_bootstrap.cc:492] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481: No bootstrap required, opened a new log
I20260927 11:18:09.927661 2711 ts_tablet_manager.cc:1397] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:09.927838 2714 tablet_bootstrap.cc:654] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:09.927872 2710 tablet_bootstrap.cc:492] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947: No bootstrap required, opened a new log
I20260927 11:18:09.927825 2711 raft_consensus.cc:348] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:09.927901 2711 raft_consensus.cc:374] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:09.927913 2710 ts_tablet_manager.cc:1397] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:09.927924 2711 raft_consensus.cc:729] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c00bb8829b4553b75bcee4fcb25481, State: Initialized, Role: FOLLOWER
I20260927 11:18:09.927980 2711 consensus_queue.cc:260] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:09.928016 2710 raft_consensus.cc:348] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:09.928078 2710 raft_consensus.cc:374] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:09.928076 2711 ts_tablet_manager.cc:1428] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:09.928099 2710 raft_consensus.cc:729] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f08c903b69472c872448f912869947, State: Initialized, Role: FOLLOWER
I20260927 11:18:09.928144 2710 consensus_queue.cc:260] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:09.928310 2710 ts_tablet_manager.cc:1428] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:09.928692 2714 tablet_bootstrap.cc:492] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841: No bootstrap required, opened a new log
I20260927 11:18:09.928745 2714 ts_tablet_manager.cc:1397] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:09.928982 2714 raft_consensus.cc:348] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:09.929093 2714 raft_consensus.cc:374] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:09.929119 2714 raft_consensus.cc:729] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35d08ffcc5148fd84df4c665d8bb841, State: Initialized, Role: FOLLOWER
I20260927 11:18:09.929163 2714 consensus_queue.cc:260] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:09.929349 2714 ts_tablet_manager.cc:1428] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:10.350173 2687 raft_consensus.cc:482] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:10.350323 2687 raft_consensus.cc:504] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:10.350553 2687 leader_election.cc:288] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7f08c903b69472c872448f912869947 (127.0.98.1:40711), a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665)
I20260927 11:18:10.350780 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e286e0321de64f33b037838bbed82a12" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7f08c903b69472c872448f912869947" is_pre_election: true
I20260927 11:18:10.350807 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e286e0321de64f33b037838bbed82a12" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35d08ffcc5148fd84df4c665d8bb841" is_pre_election: true
I20260927 11:18:10.350890 2558 raft_consensus.cc:2454] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 0.
I20260927 11:18:10.350890 2277 raft_consensus.cc:2454] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 0.
I20260927 11:18:10.351115 2352 leader_election.cc:302] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c00bb8829b4553b75bcee4fcb25481, a35d08ffcc5148fd84df4c665d8bb841; no voters:
I20260927 11:18:10.351253 2687 raft_consensus.cc:2781] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:10.351301 2687 raft_consensus.cc:482] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:10.351329 2687 raft_consensus.cc:3037] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:10.369908 2687 raft_consensus.cc:504] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:10.370124 2687 leader_election.cc:288] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 election: Requested vote from peers c7f08c903b69472c872448f912869947 (127.0.98.1:40711), a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665)
I20260927 11:18:10.370326 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e286e0321de64f33b037838bbed82a12" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7f08c903b69472c872448f912869947"
I20260927 11:18:10.370376 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e286e0321de64f33b037838bbed82a12" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35d08ffcc5148fd84df4c665d8bb841"
I20260927 11:18:10.370455 2277 raft_consensus.cc:3037] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:10.370467 2558 raft_consensus.cc:3037] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:10.371091 2558 raft_consensus.cc:2454] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 1.
I20260927 11:18:10.371351 2352 leader_election.cc:302] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c00bb8829b4553b75bcee4fcb25481, a35d08ffcc5148fd84df4c665d8bb841; no voters:
I20260927 11:18:10.371464 2687 raft_consensus.cc:2781] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:10.371515 2687 raft_consensus.cc:686] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [term 1 LEADER]: Becoming Leader. State: Replica: 78c00bb8829b4553b75bcee4fcb25481, State: Running, Role: LEADER
I20260927 11:18:10.371556 2277 raft_consensus.cc:2454] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 1.
I20260927 11:18:10.371572 2687 consensus_queue.cc:237] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:10.372262 2119 catalog_manager.cc:5495] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c00bb8829b4553b75bcee4fcb25481 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "78c00bb8829b4553b75bcee4fcb25481" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } health_report { overall_health: UNKNOWN } } }
I20260927 11:18:10.481021 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:10.478612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48848 (local address 127.0.98.2:46315)
0927 11:18:10.478709 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:10.478711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:10.478746 (+ 35us) server_negotiation.cc:407] Connection header received
0927 11:18:10.478823 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:10.478824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:10.478838 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:10.478901 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:10.479025 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:10.479511 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:10.480288 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:10.480559 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:10.480629 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:10.480771 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:10.480807 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:10.480855 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:10.480910 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 11:18:10.480932 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":55,"threads_started":1}
I20260927 11:18:10.483369 2558 raft_consensus.cc:1261] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Refusing update from remote peer 78c00bb8829b4553b75bcee4fcb25481: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:10.483398 2277 raft_consensus.cc:1261] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Refusing update from remote peer 78c00bb8829b4553b75bcee4fcb25481: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:10.483592 2687 consensus_queue.cc:1035] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:10.483716 2722 consensus_queue.cc:1035] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:10.492944 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:10.493033 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:10.493639 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78: Sending DeleteTablet for 3 replicas of tablet e286e0321de64f33b037838bbed82a12
I20260927 11:18:10.493932 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet e286e0321de64f33b037838bbed82a12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51010
I20260927 11:18:10.493965 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet e286e0321de64f33b037838bbed82a12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56474
I20260927 11:18:10.493961 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet e286e0321de64f33b037838bbed82a12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48812
I20260927 11:18:10.494184 2736 tablet_replica.cc:330] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947: stopping tablet replica
I20260927 11:18:10.494225 2738 tablet_replica.cc:330] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481: stopping tablet replica
I20260927 11:18:10.494270 2736 raft_consensus.cc:2229] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:10.494284 2738 raft_consensus.cc:2229] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:10.494313 2736 raft_consensus.cc:2258] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:10.494338 2738 raft_consensus.cc:2258] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:10.494215 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:10.494485 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:10.494225 2737 tablet_replica.cc:330] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841: stopping tablet replica
I20260927 11:18:10.494609 2736 ts_tablet_manager.cc:1905] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:10.494726 2737 raft_consensus.cc:2229] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:10.494831 2737 raft_consensus.cc:2258] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:10.494828 2738 ts_tablet_manager.cc:1905] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:10.495110 2737 ts_tablet_manager.cc:1905] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:10.495724 2258 tablet_service.cc:1467] Processing CreateTablet for tablet b5271e16c134498cb1332615bd4b261c (DEFAULT_TABLE table=test-table [id=5136d94dc7444ee9b31a642c657d5ea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:10.495723 2538 tablet_service.cc:1467] Processing CreateTablet for tablet b5271e16c134498cb1332615bd4b261c (DEFAULT_TABLE table=test-table [id=5136d94dc7444ee9b31a642c657d5ea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:10.495841 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5271e16c134498cb1332615bd4b261c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:10.495848 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5271e16c134498cb1332615bd4b261c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:10.496485 2397 tablet_service.cc:1467] Processing CreateTablet for tablet b5271e16c134498cb1332615bd4b261c (DEFAULT_TABLE table=test-table [id=5136d94dc7444ee9b31a642c657d5ea8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:10.496603 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5271e16c134498cb1332615bd4b261c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:10.496987 2739 tablet_bootstrap.cc:492] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947: Bootstrap starting.
I20260927 11:18:10.497376 2739 tablet_bootstrap.cc:654] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:10.497920 2739 tablet_bootstrap.cc:492] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947: No bootstrap required, opened a new log
I20260927 11:18:10.497959 2739 ts_tablet_manager.cc:1397] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:10.498080 2739 raft_consensus.cc:348] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:10.498145 2741 tablet_bootstrap.cc:492] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481: Bootstrap starting.
I20260927 11:18:10.498162 2739 raft_consensus.cc:374] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:10.498184 2739 raft_consensus.cc:729] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f08c903b69472c872448f912869947, State: Initialized, Role: FOLLOWER
I20260927 11:18:10.498245 2739 consensus_queue.cc:260] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:10.498364 2739 ts_tablet_manager.cc:1428] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:10.498459 2741 tablet_bootstrap.cc:654] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:10.498973 2741 tablet_bootstrap.cc:492] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481: No bootstrap required, opened a new log
I20260927 11:18:10.499043 2741 ts_tablet_manager.cc:1397] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:10.499157 2741 raft_consensus.cc:348] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:10.499225 2741 raft_consensus.cc:374] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:10.499239 2741 raft_consensus.cc:729] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c00bb8829b4553b75bcee4fcb25481, State: Initialized, Role: FOLLOWER
I20260927 11:18:10.499305 2741 consensus_queue.cc:260] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:10.499396 2741 ts_tablet_manager.cc:1428] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:10.564033 2738 ts_tablet_manager.cc:1918] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:10.564033 2737 ts_tablet_manager.cc:1918] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:10.564127 2738 log.cc:1196] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/e286e0321de64f33b037838bbed82a12
I20260927 11:18:10.564126 2737 log.cc:1196] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/e286e0321de64f33b037838bbed82a12
I20260927 11:18:10.564208 2736 ts_tablet_manager.cc:1918] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:10.564276 2736 log.cc:1196] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-0/wal/wals/e286e0321de64f33b037838bbed82a12
I20260927 11:18:10.564390 2738 ts_tablet_manager.cc:1939] T e286e0321de64f33b037838bbed82a12 P 78c00bb8829b4553b75bcee4fcb25481: Deleting consensus metadata
I20260927 11:18:10.564390 2737 ts_tablet_manager.cc:1939] T e286e0321de64f33b037838bbed82a12 P a35d08ffcc5148fd84df4c665d8bb841: Deleting consensus metadata
I20260927 11:18:10.564599 2736 ts_tablet_manager.cc:1939] T e286e0321de64f33b037838bbed82a12 P c7f08c903b69472c872448f912869947: Deleting consensus metadata
I20260927 11:18:10.565862 2743 tablet_bootstrap.cc:492] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841: Bootstrap starting.
I20260927 11:18:10.566282 2743 tablet_bootstrap.cc:654] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:10.567374 2743 tablet_bootstrap.cc:492] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841: No bootstrap required, opened a new log
I20260927 11:18:10.567446 2743 ts_tablet_manager.cc:1397] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:10.567633 2743 raft_consensus.cc:348] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:10.567718 2743 raft_consensus.cc:374] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:10.567746 2743 raft_consensus.cc:729] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35d08ffcc5148fd84df4c665d8bb841, State: Initialized, Role: FOLLOWER
I20260927 11:18:10.567806 2743 consensus_queue.cc:260] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:10.567917 2743 ts_tablet_manager.cc:1428] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:10.570003 2106 catalog_manager.cc:4841] TS c7f08c903b69472c872448f912869947 (127.0.98.1:40711): tablet e286e0321de64f33b037838bbed82a12 (table test-table [id=97e8c21aa9534659913201ba2e1d52df]) successfully deleted
I20260927 11:18:10.570104 2108 catalog_manager.cc:4841] TS 78c00bb8829b4553b75bcee4fcb25481 (127.0.98.2:46315): tablet e286e0321de64f33b037838bbed82a12 (table test-table [id=97e8c21aa9534659913201ba2e1d52df]) successfully deleted
I20260927 11:18:10.570155 2108 catalog_manager.cc:4841] TS a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665): tablet e286e0321de64f33b037838bbed82a12 (table test-table [id=97e8c21aa9534659913201ba2e1d52df]) successfully deleted
I20260927 11:18:10.576292 2687 raft_consensus.cc:482] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:10.576371 2687 raft_consensus.cc:504] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:10.576550 2687 leader_election.cc:288] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7f08c903b69472c872448f912869947 (127.0.98.1:40711), a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665)
I20260927 11:18:10.576748 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5271e16c134498cb1332615bd4b261c" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35d08ffcc5148fd84df4c665d8bb841" is_pre_election: true
I20260927 11:18:10.576761 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5271e16c134498cb1332615bd4b261c" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7f08c903b69472c872448f912869947" is_pre_election: true
I20260927 11:18:10.576840 2558 raft_consensus.cc:2454] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 0.
I20260927 11:18:10.576848 2277 raft_consensus.cc:2454] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 0.
I20260927 11:18:10.577025 2352 leader_election.cc:302] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c00bb8829b4553b75bcee4fcb25481, a35d08ffcc5148fd84df4c665d8bb841; no voters:
I20260927 11:18:10.577164 2687 raft_consensus.cc:2781] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:10.577217 2687 raft_consensus.cc:482] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:10.577248 2687 raft_consensus.cc:3037] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:10.577883 2687 raft_consensus.cc:504] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:10.578018 2687 leader_election.cc:288] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 election: Requested vote from peers c7f08c903b69472c872448f912869947 (127.0.98.1:40711), a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665)
I20260927 11:18:10.578166 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5271e16c134498cb1332615bd4b261c" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a35d08ffcc5148fd84df4c665d8bb841"
I20260927 11:18:10.578179 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5271e16c134498cb1332615bd4b261c" candidate_uuid: "78c00bb8829b4553b75bcee4fcb25481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7f08c903b69472c872448f912869947"
I20260927 11:18:10.578245 2558 raft_consensus.cc:3037] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:10.578251 2277 raft_consensus.cc:3037] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:10.578800 2558 raft_consensus.cc:2454] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 1.
I20260927 11:18:10.578847 2277 raft_consensus.cc:2454] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78c00bb8829b4553b75bcee4fcb25481 in term 1.
I20260927 11:18:10.578992 2352 leader_election.cc:302] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78c00bb8829b4553b75bcee4fcb25481, a35d08ffcc5148fd84df4c665d8bb841; no voters:
I20260927 11:18:10.579159 2687 raft_consensus.cc:2781] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:10.579216 2687 raft_consensus.cc:686] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [term 1 LEADER]: Becoming Leader. State: Replica: 78c00bb8829b4553b75bcee4fcb25481, State: Running, Role: LEADER
I20260927 11:18:10.579265 2687 consensus_queue.cc:237] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } }
I20260927 11:18:10.579857 2120 catalog_manager.cc:5495] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78c00bb8829b4553b75bcee4fcb25481 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "78c00bb8829b4553b75bcee4fcb25481" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78c00bb8829b4553b75bcee4fcb25481" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 } health_report { overall_health: UNKNOWN } } }
I20260927 11:18:10.592712 2558 raft_consensus.cc:1261] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Refusing update from remote peer 78c00bb8829b4553b75bcee4fcb25481: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:10.592712 2277 raft_consensus.cc:1261] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947 [term 1 FOLLOWER]: Refusing update from remote peer 78c00bb8829b4553b75bcee4fcb25481: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:10.592974 2687 consensus_queue.cc:1035] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a35d08ffcc5148fd84df4c665d8bb841" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:10.593092 2722 consensus_queue.cc:1035] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7f08c903b69472c872448f912869947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 11:18:10.597532 2719 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 11:18:12.114563 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:12.114669 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:12.115448 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82b70d7bb3d841b28899e4dd6432be78: Sending DeleteTablet for 3 replicas of tablet b5271e16c134498cb1332615bd4b261c
I20260927 11:18:12.115756 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet b5271e16c134498cb1332615bd4b261c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48812
I20260927 11:18:12.115757 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet b5271e16c134498cb1332615bd4b261c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51010
I20260927 11:18:12.115911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 2193
I20260927 11:18:12.115944 2767 tablet_replica.cc:330] T b5271e16c134498cb1332615bd4b261c P c7f08c903b69472c872448f912869947: stopping tablet replica
I20260927 11:18:12.116382 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet b5271e16c134498cb1332615bd4b261c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56474
I20260927 11:18:12.116451 2768 tablet_replica.cc:330] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481: stopping tablet replica
I20260927 11:18:12.116508 2768 raft_consensus.cc:2229] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:12.116577 2768 raft_consensus.cc:2258] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:12.116605 2769 tablet_replica.cc:330] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841: stopping tablet replica
I20260927 11:18:12.116657 2769 raft_consensus.cc:2229] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:12.116689 2769 raft_consensus.cc:2258] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:12.116909 2769 ts_tablet_manager.cc:1905] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:12.116909 2768 ts_tablet_manager.cc:1905] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:12.118721 2768 ts_tablet_manager.cc:1918] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 11:18:12.118722 2769 ts_tablet_manager.cc:1918] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 11:18:12.118782 2768 log.cc:1196] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/b5271e16c134498cb1332615bd4b261c
I20260927 11:18:12.118782 2769 log.cc:1196] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/b5271e16c134498cb1332615bd4b261c
I20260927 11:18:12.119037 2769 ts_tablet_manager.cc:1939] T b5271e16c134498cb1332615bd4b261c P a35d08ffcc5148fd84df4c665d8bb841: Deleting consensus metadata
I20260927 11:18:12.119029 2768 ts_tablet_manager.cc:1939] T b5271e16c134498cb1332615bd4b261c P 78c00bb8829b4553b75bcee4fcb25481: Deleting consensus metadata
I20260927 11:18:12.119313 2108 catalog_manager.cc:4841] TS a35d08ffcc5148fd84df4c665d8bb841 (127.0.98.3:38665): tablet b5271e16c134498cb1332615bd4b261c (table test-table [id=5136d94dc7444ee9b31a642c657d5ea8]) successfully deleted
I20260927 11:18:12.119407 2108 catalog_manager.cc:4841] TS 78c00bb8829b4553b75bcee4fcb25481 (127.0.98.2:46315): tablet b5271e16c134498cb1332615bd4b261c (table test-table [id=5136d94dc7444ee9b31a642c657d5ea8]) successfully deleted
W20260927 11:18:12.123404 2106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40711 (error 108)
I20260927 11:18:12.123675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 2333
I20260927 11:18:12.124487 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:12.123773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40711 (local address 127.0.0.1:51036)
0927 11:18:12.124386 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:12.124440 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":62,"threads_started":1}
W20260927 11:18:12.124636 2106 catalog_manager.cc:4584] TS c7f08c903b69472c872448f912869947 (127.0.98.1:40711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5271e16c134498cb1332615bd4b261c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40711: connect: Connection refused (error 111)
I20260927 11:18:12.130015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 2473
I20260927 11:18:12.134668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 2089
2026-09-27T11:18:12Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5979 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:12 dist-test-slave-lsvg krb5kdc[2774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:12 dist-test-slave-lsvg krb5kdc[2774](info): set up 2 sockets
Sep 27 11:18:12 dist-test-slave-lsvg krb5kdc[2774](info): commencing operation
krb5kdc: starting...
W20260927 11:18:13.168888 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:13 dist-test-slave-lsvg krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:18:13.288988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:13.368813 2787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:13.369019 2787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:13.369041 2787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:13.370253 2787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:13.370304 2787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:13.370319 2787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:13.370332 2787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:13.370344 2787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:13.371845 2787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:13.372080 2787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:13.372315 2787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:13.377386 2792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:13.377516 2793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:13.377421 2795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:13.377581 2787 server_base.cc:1034] running on GCE node
I20260927 11:18:13.377848 2787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:13.378121 2787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:13.379276 2787 hybrid_clock.cc:648] HybridClock initialized: now 1790507893379269 us; error 38 us; skew 500 ppm
Sep 27 11:18:13 dist-test-slave-lsvg krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:13.381779 2787 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:18:13.382184 2787 webserver.cc:460] Webserver started at http://127.0.98.62:38189/ using document root <none> and password file <none>
I20260927 11:18:13.382416 2787 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:13.382483 2787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:13.382635 2787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:13.383692 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "1f068c39c8d0495fb73846bb8f164de2"
format_stamp: "Formatted at 2026-09-27 11:18:13 on dist-test-slave-lsvg"
server_key: "765b7070d508d32915f27c61652f4700"
server_key_iv: "712c2647e5a756e0a10888b0998a76ed"
server_key_version: "encryptionkey@0"
I20260927 11:18:13.384039 2787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f068c39c8d0495fb73846bb8f164de2"
format_stamp: "Formatted at 2026-09-27 11:18:13 on dist-test-slave-lsvg"
server_key: "765b7070d508d32915f27c61652f4700"
server_key_iv: "712c2647e5a756e0a10888b0998a76ed"
server_key_version: "encryptionkey@0"
I20260927 11:18:13.385490 2787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:13.386307 2802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:13.386564 2787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:13.386662 2787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "1f068c39c8d0495fb73846bb8f164de2"
format_stamp: "Formatted at 2026-09-27 11:18:13 on dist-test-slave-lsvg"
server_key: "765b7070d508d32915f27c61652f4700"
server_key_iv: "712c2647e5a756e0a10888b0998a76ed"
server_key_version: "encryptionkey@0"
I20260927 11:18:13.386750 2787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:13.405196 2787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:13.407408 2787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:13.407562 2787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:13.411355 2787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39111
I20260927 11:18:13.411408 2854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39111 every 8 connection(s)
I20260927 11:18:13.411904 2787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:13.412563 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:13.413703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 2787
I20260927 11:18:13.413800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:13.413975 392 external_mini_cluster.cc:1428] Setting key 5c715a5aff22f9033fd8564b4f056d2a
I20260927 11:18:13.415146 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2: Bootstrap starting.
I20260927 11:18:13.415853 2855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:13.416193 2855 log.cc:824] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:13.416841 2855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2: No bootstrap required, opened a new log
I20260927 11:18:13.418601 2855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f068c39c8d0495fb73846bb8f164de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } }
I20260927 11:18:13.418732 2855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:13.418754 2855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f068c39c8d0495fb73846bb8f164de2, State: Initialized, Role: FOLLOWER
I20260927 11:18:13.418905 2855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f068c39c8d0495fb73846bb8f164de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } }
I20260927 11:18:13.418982 2855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:13.419020 2855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:13.419067 2855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:13.419723 2855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f068c39c8d0495fb73846bb8f164de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } }
I20260927 11:18:13.419835 2855 leader_election.cc:302] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f068c39c8d0495fb73846bb8f164de2; no voters:
I20260927 11:18:13.420004 2855 leader_election.cc:288] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:13.420047 2860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:13.420202 2860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [term 1 LEADER]: Becoming Leader. State: Replica: 1f068c39c8d0495fb73846bb8f164de2, State: Running, Role: LEADER
I20260927 11:18:13.420291 2855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:13.420315 2860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f068c39c8d0495fb73846bb8f164de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } }
I20260927 11:18:13.420701 2862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f068c39c8d0495fb73846bb8f164de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f068c39c8d0495fb73846bb8f164de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } } }
I20260927 11:18:13.420786 2862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:13.420679 2860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f068c39c8d0495fb73846bb8f164de2. Latest consensus state: current_term: 1 leader_uuid: "1f068c39c8d0495fb73846bb8f164de2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f068c39c8d0495fb73846bb8f164de2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } } }
I20260927 11:18:13.420994 2860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:13.421135 2866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 11:18:13 dist-test-slave-lsvg krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:13.421587 2866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:13.423684 2866 catalog_manager.cc:1347] Generated new cluster ID: a96f9a51bd724381bb74b6c8e6659292
I20260927 11:18:13.423738 2866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:13.425560 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:13.414480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.0.98.62:39111)
0927 11:18:13.414660 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:13.414668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:13.414691 (+ 23us) server_negotiation.cc:407] Connection header received
0927 11:18:13.415095 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:13.415101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:13.415144 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:13.415295 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:13.416101 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.417096 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.418161 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.419142 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.419226 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:13.422201 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:13.422206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:13.422206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:13.422218 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:13.424125 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:13.424463 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:13.424466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:13.424467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:13.424556 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:13.424756 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:13.424757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:13.424757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:13.424984 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:13.425162 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:13.425254 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 11:18:13.425386 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260927 11:18:13.450954 2866 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:13.451524 2866 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:13.464429 2866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f068c39c8d0495fb73846bb8f164de2: Generated new TSK 0
I20260927 11:18:13.464646 2866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:18:13.500774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39111
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:13.583400 2881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:13.583588 2881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:13.583611 2881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:13.584769 2881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:13.584811 2881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:13.584862 2881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:13.586459 2881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:13.586690 2881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:13.586949 2881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:13.591918 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:13.591929 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:13.591938 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:13.592093 2881 server_base.cc:1034] running on GCE node
I20260927 11:18:13.592470 2881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:13.592686 2881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:13.593827 2881 hybrid_clock.cc:648] HybridClock initialized: now 1790507893593816 us; error 26 us; skew 500 ppm
Sep 27 11:18:13 dist-test-slave-lsvg krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:13.596414 2881 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:18:13.596865 2881 webserver.cc:460] Webserver started at http://127.0.98.1:45663/ using document root <none> and password file <none>
I20260927 11:18:13.597141 2881 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:13.597190 2881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:13.597328 2881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:13.598343 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "0c733f4f864147ba9b80128e53818ecb"
format_stamp: "Formatted at 2026-09-27 11:18:13 on dist-test-slave-lsvg"
server_key: "e55dc6aca142c5067bc57696aac4278a"
server_key_iv: "8fa4a376b3acab7577384b9721789d09"
server_key_version: "encryptionkey@0"
I20260927 11:18:13.598682 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0c733f4f864147ba9b80128e53818ecb"
format_stamp: "Formatted at 2026-09-27 11:18:13 on dist-test-slave-lsvg"
server_key: "e55dc6aca142c5067bc57696aac4278a"
server_key_iv: "8fa4a376b3acab7577384b9721789d09"
server_key_version: "encryptionkey@0"
I20260927 11:18:13.599941 2881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:13.600646 2896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:13.600818 2881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:13.600894 2881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "0c733f4f864147ba9b80128e53818ecb"
format_stamp: "Formatted at 2026-09-27 11:18:13 on dist-test-slave-lsvg"
server_key: "e55dc6aca142c5067bc57696aac4278a"
server_key_iv: "8fa4a376b3acab7577384b9721789d09"
server_key_version: "encryptionkey@0"
I20260927 11:18:13.600955 2881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:13.615428 2881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:13.617405 2881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:13.617515 2881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:13.617699 2881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:13.618007 2881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:13.618041 2881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:13.618064 2881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:13.618108 2881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:13.623962 2881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40911
I20260927 11:18:13.624039 3009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40911 every 8 connection(s)
I20260927 11:18:13.624353 2881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:18:13.626030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 2881
I20260927 11:18:13.626109 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:13.626286 392 external_mini_cluster.cc:1428] Setting key cf77ec868b68ef2c51ef5cbc80ee0da0
Sep 27 11:18:13 dist-test-slave-lsvg krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:13.634310 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:13.625101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39111 (local address 127.0.98.1:35047)
0927 11:18:13.625275 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:13.625311 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:13.626374 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:13.626632 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:13.626636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:13.626903 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:13.627092 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:13.627096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.627783 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.627784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:13.628600 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:13.628602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.629115 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.629116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:13.629232 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:13.629773 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:13.629781 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:13.631152 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:13.632967 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:13.632973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:13.632975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:13.633245 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:13.633457 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:13.633457 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:13.633458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:13.633551 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:13.633841 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:13.633844 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:13.633957 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 11:18:13.634029 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 11:18:13.634145 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260927 11:18:13.634315 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:13.625113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35047 (local address 127.0.98.62:39111)
0927 11:18:13.625156 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:13.625158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:13.625413 (+ 255us) server_negotiation.cc:407] Connection header received
0927 11:18:13.626479 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:13.626480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:13.626493 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:13.626545 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:13.627194 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.627695 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.628689 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.628993 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.629092 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:13.631244 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:13.631245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:13.631246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:13.631256 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:13.632887 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:13.633315 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:13.633317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:13.633317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:13.633383 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:13.633621 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:13.633622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:13.633623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:13.633772 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:13.633848 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:13.634094 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 11:18:13.634194 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 11:18:13.634871 3010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39111
I20260927 11:18:13.634980 3010 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:13.635174 3010 heartbeater.cc:511] Master 127.0.98.62:39111 requested a full tablet report, sending...
I20260927 11:18:13.635615 2819 ts_manager.cc:194] Registered new tserver with Master: 0c733f4f864147ba9b80128e53818ecb (127.0.98.1:40911)
I20260927 11:18:13.636296 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:18:13.652257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39111
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:13.733393 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:13.733630 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:13.733672 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:13.734864 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:13.734927 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:13.734974 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:13.736552 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:13.736810 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:13.737066 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:13.741950 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:13.741981 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:13.741981 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:13.742168 3015 server_base.cc:1034] running on GCE node
I20260927 11:18:13.742408 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:13.742633 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:13.743767 3015 hybrid_clock.cc:648] HybridClock initialized: now 1790507893743752 us; error 32 us; skew 500 ppm
Sep 27 11:18:13 dist-test-slave-lsvg krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:13.746301 3015 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:18:13.746713 3015 webserver.cc:460] Webserver started at http://127.0.98.2:45689/ using document root <none> and password file <none>
I20260927 11:18:13.746951 3015 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:13.747021 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:13.747164 3015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:13.748217 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "46cf898169ba4c3aa89c8a1f07a108fa"
format_stamp: "Formatted at 2026-09-27 11:18:13 on dist-test-slave-lsvg"
server_key: "cc1d3393a637b47e1f7722ea75517400"
server_key_iv: "3b702a14d65875b98ae2d057ac812f53"
server_key_version: "encryptionkey@0"
I20260927 11:18:13.748562 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "46cf898169ba4c3aa89c8a1f07a108fa"
format_stamp: "Formatted at 2026-09-27 11:18:13 on dist-test-slave-lsvg"
server_key: "cc1d3393a637b47e1f7722ea75517400"
server_key_iv: "3b702a14d65875b98ae2d057ac812f53"
server_key_version: "encryptionkey@0"
I20260927 11:18:13.749913 3015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:13.750582 3030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:13.750744 3015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:13.750840 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "46cf898169ba4c3aa89c8a1f07a108fa"
format_stamp: "Formatted at 2026-09-27 11:18:13 on dist-test-slave-lsvg"
server_key: "cc1d3393a637b47e1f7722ea75517400"
server_key_iv: "3b702a14d65875b98ae2d057ac812f53"
server_key_version: "encryptionkey@0"
I20260927 11:18:13.750913 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:13.761013 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:13.763289 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:13.763450 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:13.763715 3015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:13.764026 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:13.764084 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:13.764127 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:13.764168 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:13.769887 3015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43475
I20260927 11:18:13.769934 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43475 every 8 connection(s)
I20260927 11:18:13.770262 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:18:13 dist-test-slave-lsvg krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:13.776913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 3015
I20260927 11:18:13.777016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:13.777267 392 external_mini_cluster.cc:1428] Setting key e63719b98c1d9e54355d08c05f7b5e2a
I20260927 11:18:13.779795 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:13.770954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51245 (local address 127.0.98.62:39111)
0927 11:18:13.771023 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:13.771024 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:13.771200 (+ 176us) server_negotiation.cc:407] Connection header received
0927 11:18:13.772091 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:13.772092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:13.772101 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:13.772180 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:13.772738 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.773291 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.774237 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.774416 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.774465 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:13.776516 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:13.776517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:13.776517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:13.776523 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:13.778205 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:13.778718 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:13.778719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:13.778719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:13.778773 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:13.779052 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:13.779053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:13.779053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:13.779217 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:13.779314 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:13.779590 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 11:18:13.779682 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 11:18:13.779861 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:13.770931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39111 (local address 127.0.98.2:51245)
0927 11:18:13.771085 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:13.771125 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:13.772005 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:13.772236 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:13.772239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:13.772483 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:13.772670 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:13.772675 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.773376 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.773377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:13.774170 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:13.774171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.774481 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.774481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:13.774593 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:13.775099 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:13.775107 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:13.776431 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:13.778283 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:13.778290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:13.778293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:13.778634 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:13.778845 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:13.778846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:13.778846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:13.778989 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:13.779294 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:13.779301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:13.779461 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 11:18:13.779548 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 11:18:13.779673 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260927 11:18:13.780345 3144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39111
I20260927 11:18:13.780427 3144 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:13.780568 3144 heartbeater.cc:511] Master 127.0.98.62:39111 requested a full tablet report, sending...
I20260927 11:18:13.780928 2819 ts_manager.cc:194] Registered new tserver with Master: 46cf898169ba4c3aa89c8a1f07a108fa (127.0.98.2:43475)
I20260927 11:18:13.781333 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51245
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:18:13.804904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39111
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:13.886767 3149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:13.887015 3149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:13.887056 3149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:13.888356 3149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:13.888432 3149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:13.888476 3149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:13.890167 3149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:13.890441 3149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:13.890669 3149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:13.895756 3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:13.895938 3149 server_base.cc:1034] running on GCE node
W20260927 11:18:13.895762 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:13.895828 3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:13.896234 3149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:13.896498 3149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:13.897657 3149 hybrid_clock.cc:648] HybridClock initialized: now 1790507893897653 us; error 37 us; skew 500 ppm
Sep 27 11:18:13 dist-test-slave-lsvg krb5kdc[2774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:13.900121 3149 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:18:13.900576 3149 webserver.cc:460] Webserver started at http://127.0.98.3:45909/ using document root <none> and password file <none>
I20260927 11:18:13.900764 3149 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:13.900804 3149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:13.900895 3149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:13.901939 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "40f0d1b33877465684ddf2880446cf31"
format_stamp: "Formatted at 2026-09-27 11:18:13 on dist-test-slave-lsvg"
server_key: "058db72434797d41cea994721d8993f7"
server_key_iv: "8c61e769bae33cde37206b989af9a5b6"
server_key_version: "encryptionkey@0"
I20260927 11:18:13.902235 3149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "40f0d1b33877465684ddf2880446cf31"
format_stamp: "Formatted at 2026-09-27 11:18:13 on dist-test-slave-lsvg"
server_key: "058db72434797d41cea994721d8993f7"
server_key_iv: "8c61e769bae33cde37206b989af9a5b6"
server_key_version: "encryptionkey@0"
I20260927 11:18:13.903481 3149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:18:13.904182 3164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:13.904361 3149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:13.904453 3149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "40f0d1b33877465684ddf2880446cf31"
format_stamp: "Formatted at 2026-09-27 11:18:13 on dist-test-slave-lsvg"
server_key: "058db72434797d41cea994721d8993f7"
server_key_iv: "8c61e769bae33cde37206b989af9a5b6"
server_key_version: "encryptionkey@0"
I20260927 11:18:13.904516 3149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:13.933540 3149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:13.935909 3149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:13.936028 3149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:13.936220 3149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:13.936555 3149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:13.936592 3149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:13.936635 3149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:13.936661 3149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:13.942369 3149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37323
I20260927 11:18:13.942448 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37323 every 8 connection(s)
I20260927 11:18:13.942764 3149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 11:18:13 dist-test-slave-lsvg krb5kdc[2774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:13.950002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 3149
I20260927 11:18:13.950117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:13.950315 392 external_mini_cluster.cc:1428] Setting key 2fa79d0e1e53576be483be5837a3b9dd
I20260927 11:18:13.952522 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:13.943572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44091 (local address 127.0.98.62:39111)
0927 11:18:13.943618 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:13.943620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:13.943788 (+ 168us) server_negotiation.cc:407] Connection header received
0927 11:18:13.944711 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:13.944711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:13.944720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:13.944784 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:13.945480 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.945977 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.946926 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.947106 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.947156 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:13.949337 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:13.949339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:13.949340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:13.949349 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:13.950937 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:13.951581 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:13.951582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:13.951582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:13.951639 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:13.951989 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:13.951990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:13.951990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:13.952104 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:13.952161 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:13.952398 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 11:18:13.952458 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:18:13.952775 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:13.943518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39111 (local address 127.0.98.3:44091)
0927 11:18:13.943665 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:13.943707 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:13.944624 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:13.944857 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:13.944861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:13.945189 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:13.945404 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:13.945409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.946050 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.946051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:13.946855 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:13.946857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:13.947187 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:13.947188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:13.947292 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:13.947805 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:13.947814 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:13.949242 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:13.951027 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:13.951033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:13.951034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:13.951494 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:13.951719 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:13.951720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:13.951720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:13.951907 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:13.952163 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:13.952167 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:13.952289 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 11:18:13.952375 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 11:18:13.952497 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260927 11:18:13.953316 3278 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39111
I20260927 11:18:13.953433 3278 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:13.953650 3278 heartbeater.cc:511] Master 127.0.98.62:39111 requested a full tablet report, sending...
I20260927 11:18:13.954005 2819 ts_manager.cc:194] Registered new tserver with Master: 40f0d1b33877465684ddf2880446cf31 (127.0.98.3:37323)
I20260927 11:18:13.954380 2819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44091
I20260927 11:18:13.961935 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 11:18:13.973093 2858 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:13.972111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45284 (local address 127.0.98.62:39111)
0927 11:18:13.972186 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:13.972188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:13.972281 (+ 93us) server_negotiation.cc:407] Connection header received
0927 11:18:13.972362 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:13.972363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:13.972373 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:13.972441 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:13.972998 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45284: BlockingRecv error: recv got EOF from 127.0.0.1:45284 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 11:18:13.974047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 2881
I20260927 11:18:13.979091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 3015
I20260927 11:18:13.983348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 3149
I20260927 11:18:13.987381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 2787
2026-09-27T11:18:13Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1851 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:14 dist-test-slave-lsvg krb5kdc[3290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:14 dist-test-slave-lsvg krb5kdc[3290](info): set up 2 sockets
Sep 27 11:18:14 dist-test-slave-lsvg krb5kdc[3290](info): commencing operation
krb5kdc: starting...
W20260927 11:18:15.036892 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:15 dist-test-slave-lsvg krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:18:15.160243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38073
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:15.241675 3303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:15.241947 3303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:15.241990 3303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:15.243234 3303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:15.243314 3303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:15.243338 3303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:15.243358 3303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:15.243391 3303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:15.244951 3303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:15.245265 3303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:15.245507 3303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:15.250559 3311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:15.250555 3309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:15.250742 3303 server_base.cc:1034] running on GCE node
W20260927 11:18:15.250639 3308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:15.251073 3303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:15.251334 3303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:15.252496 3303 hybrid_clock.cc:648] HybridClock initialized: now 1790507895252479 us; error 37 us; skew 500 ppm
Sep 27 11:18:15 dist-test-slave-lsvg krb5kdc[3290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:15.255230 3303 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:18:15.255707 3303 webserver.cc:460] Webserver started at http://127.0.98.62:44531/ using document root <none> and password file <none>
I20260927 11:18:15.255965 3303 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:15.256012 3303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:15.256181 3303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:15.257346 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "1469a6457dd747aca3ef78db5a50cf94"
format_stamp: "Formatted at 2026-09-27 11:18:15 on dist-test-slave-lsvg"
server_key: "fc0b993ec9707afb6571526452b79344"
server_key_iv: "1fd01ff79d8b48588515232eedab419d"
server_key_version: "encryptionkey@0"
I20260927 11:18:15.257701 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "1469a6457dd747aca3ef78db5a50cf94"
format_stamp: "Formatted at 2026-09-27 11:18:15 on dist-test-slave-lsvg"
server_key: "fc0b993ec9707afb6571526452b79344"
server_key_iv: "1fd01ff79d8b48588515232eedab419d"
server_key_version: "encryptionkey@0"
I20260927 11:18:15.259047 3303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:15.259815 3318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:15.260016 3303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:15.260111 3303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "1469a6457dd747aca3ef78db5a50cf94"
format_stamp: "Formatted at 2026-09-27 11:18:15 on dist-test-slave-lsvg"
server_key: "fc0b993ec9707afb6571526452b79344"
server_key_iv: "1fd01ff79d8b48588515232eedab419d"
server_key_version: "encryptionkey@0"
I20260927 11:18:15.260196 3303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:15.271708 3303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:15.274207 3303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:15.274398 3303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:15.278482 3303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38073
I20260927 11:18:15.278517 3370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38073 every 8 connection(s)
I20260927 11:18:15.278913 3303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:15.279757 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:15.282037 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94: Bootstrap starting.
I20260927 11:18:15.282614 3371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:15.282910 3371 log.cc:824] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:15.283607 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94: No bootstrap required, opened a new log
I20260927 11:18:15.284936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 3303
I20260927 11:18:15.285051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:15.285266 392 external_mini_cluster.cc:1428] Setting key d621b314e35a50d14f5b784e789db96e
I20260927 11:18:15.285290 3371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1469a6457dd747aca3ef78db5a50cf94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38073 } }
I20260927 11:18:15.285465 3371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:15.285521 3371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1469a6457dd747aca3ef78db5a50cf94, State: Initialized, Role: FOLLOWER
I20260927 11:18:15.285665 3371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1469a6457dd747aca3ef78db5a50cf94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38073 } }
I20260927 11:18:15.285748 3371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:15.285815 3371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:15.285917 3371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:15.286587 3371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1469a6457dd747aca3ef78db5a50cf94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38073 } }
I20260927 11:18:15.286705 3371 leader_election.cc:302] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1469a6457dd747aca3ef78db5a50cf94; no voters:
I20260927 11:18:15.286873 3371 leader_election.cc:288] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:15.286943 3376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:15.287127 3371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:15.287173 3376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [term 1 LEADER]: Becoming Leader. State: Replica: 1469a6457dd747aca3ef78db5a50cf94, State: Running, Role: LEADER
I20260927 11:18:15.287397 3376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1469a6457dd747aca3ef78db5a50cf94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38073 } }
I20260927 11:18:15.287813 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1469a6457dd747aca3ef78db5a50cf94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1469a6457dd747aca3ef78db5a50cf94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38073 } } }
I20260927 11:18:15.287885 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:15.288053 3378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1469a6457dd747aca3ef78db5a50cf94. Latest consensus state: current_term: 1 leader_uuid: "1469a6457dd747aca3ef78db5a50cf94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1469a6457dd747aca3ef78db5a50cf94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38073 } } }
I20260927 11:18:15.288106 3378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:15.288203 3385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:15.288762 3385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 11:18:15 dist-test-slave-lsvg krb5kdc[3290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:15.293494 3385 catalog_manager.cc:1347] Generated new cluster ID: 6c6d37c0208d4d039b90e5c831e44b62
I20260927 11:18:15.293555 3385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:15.296898 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:15.285745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.0.98.62:38073)
0927 11:18:15.286354 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:15.286362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:15.286388 (+ 26us) server_negotiation.cc:407] Connection header received
0927 11:18:15.286488 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:15.286494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:15.286536 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:15.286725 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:15.287550 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:15.289178 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:15.290102 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:15.290535 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:15.290666 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:15.292972 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:15.292976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:15.292976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:15.292988 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:15.295265 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:15.295666 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:15.295670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:15.295670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:15.295787 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:15.295987 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:15.295989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:15.295989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:15.296297 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:15.296474 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:15.296543 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 11:18:15.296688 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":557,"thread_start_us":511,"threads_started":1}
I20260927 11:18:15.297835 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:18:15.304232 3385 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:15.304993 3385 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:15.311010 3385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1469a6457dd747aca3ef78db5a50cf94: Generated new TSK 0
I20260927 11:18:15.311204 3385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:18:15.318792 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:15.314917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.0.98.62:38073)
0927 11:18:15.314983 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:15.314985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:15.315146 (+ 161us) server_negotiation.cc:407] Connection header received
0927 11:18:15.315222 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:15.315223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:15.315235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:15.315288 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:15.315708 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:15.316228 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:15.317360 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:15.317574 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:15.317629 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:15.318311 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:15.318313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:15.318313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:15.318505 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:15.318521 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:15.318528 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:15.318573 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:15.318658 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 11:18:15.318683 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 11:18:15.321177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 3303
2026-09-27T11:18:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1331 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:15 dist-test-slave-lsvg krb5kdc[3404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:15 dist-test-slave-lsvg krb5kdc[3404](info): set up 2 sockets
Sep 27 11:18:15 dist-test-slave-lsvg krb5kdc[3404](info): commencing operation
krb5kdc: starting...
W20260927 11:18:16.365013 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:16 dist-test-slave-lsvg krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:18:16.488698 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:16.568727 3417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:16.568950 3417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:16.568979 3417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:16.570241 3417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:16.570287 3417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:16.570308 3417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:16.570365 3417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:16.570384 3417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:16.571892 3417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:16.572158 3417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:16.572394 3417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:16.577260 3425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:16.577270 3423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:16.577433 3417 server_base.cc:1034] running on GCE node
W20260927 11:18:16.577340 3422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:16.577752 3417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:16.577986 3417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:16.579119 3417 hybrid_clock.cc:648] HybridClock initialized: now 1790507896579103 us; error 39 us; skew 500 ppm
Sep 27 11:18:16 dist-test-slave-lsvg krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:16.581919 3417 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:18:16.582346 3417 webserver.cc:460] Webserver started at http://127.0.98.62:42171/ using document root <none> and password file <none>
I20260927 11:18:16.582592 3417 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:16.582660 3417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:16.582809 3417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:16.583850 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "d204da5b3d044ce18cf772cf615facfd"
format_stamp: "Formatted at 2026-09-27 11:18:16 on dist-test-slave-lsvg"
server_key: "51009cee74ff1046626acc4ba0187390"
server_key_iv: "d823b99f2aaff07cb8909b59e697a2ea"
server_key_version: "encryptionkey@0"
I20260927 11:18:16.584198 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "d204da5b3d044ce18cf772cf615facfd"
format_stamp: "Formatted at 2026-09-27 11:18:16 on dist-test-slave-lsvg"
server_key: "51009cee74ff1046626acc4ba0187390"
server_key_iv: "d823b99f2aaff07cb8909b59e697a2ea"
server_key_version: "encryptionkey@0"
I20260927 11:18:16.585584 3417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:16.586344 3432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:16.586509 3417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:16.586613 3417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "d204da5b3d044ce18cf772cf615facfd"
format_stamp: "Formatted at 2026-09-27 11:18:16 on dist-test-slave-lsvg"
server_key: "51009cee74ff1046626acc4ba0187390"
server_key_iv: "d823b99f2aaff07cb8909b59e697a2ea"
server_key_version: "encryptionkey@0"
I20260927 11:18:16.586707 3417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:16.597667 3417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:16.599586 3417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:16.599704 3417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:16.603626 3417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40741
I20260927 11:18:16.603663 3484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40741 every 8 connection(s)
I20260927 11:18:16.604012 3417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:16.604984 3485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:16.607336 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd: Bootstrap starting.
I20260927 11:18:16.607960 3485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:16.608258 3485 log.cc:824] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:16.608983 3485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd: No bootstrap required, opened a new log
I20260927 11:18:16.610631 3485 raft_consensus.cc:348] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d204da5b3d044ce18cf772cf615facfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } }
I20260927 11:18:16.610780 3485 raft_consensus.cc:374] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:16.610802 3485 raft_consensus.cc:729] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d204da5b3d044ce18cf772cf615facfd, State: Initialized, Role: FOLLOWER
I20260927 11:18:16.610947 3485 consensus_queue.cc:260] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d204da5b3d044ce18cf772cf615facfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } }
I20260927 11:18:16.611034 3485 raft_consensus.cc:388] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:16.611058 3485 raft_consensus.cc:482] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:16.611088 3485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:16.611928 3485 raft_consensus.cc:504] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d204da5b3d044ce18cf772cf615facfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } }
I20260927 11:18:16.612020 3485 leader_election.cc:302] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d204da5b3d044ce18cf772cf615facfd; no voters:
I20260927 11:18:16.612160 3485 leader_election.cc:288] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:16.612217 3488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:16.612417 3488 raft_consensus.cc:686] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [term 1 LEADER]: Becoming Leader. State: Replica: d204da5b3d044ce18cf772cf615facfd, State: Running, Role: LEADER
I20260927 11:18:16.612475 3485 sys_catalog.cc:564] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:16.612558 3488 consensus_queue.cc:237] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d204da5b3d044ce18cf772cf615facfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } }
I20260927 11:18:16.612927 3490 sys_catalog.cc:455] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d204da5b3d044ce18cf772cf615facfd. Latest consensus state: current_term: 1 leader_uuid: "d204da5b3d044ce18cf772cf615facfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d204da5b3d044ce18cf772cf615facfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } } }
I20260927 11:18:16.612922 3489 sys_catalog.cc:455] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d204da5b3d044ce18cf772cf615facfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d204da5b3d044ce18cf772cf615facfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } } }
I20260927 11:18:16.613111 3489 sys_catalog.cc:458] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:16.613101 3490 sys_catalog.cc:458] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:16.613340 3497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:16.613426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 3417
I20260927 11:18:16.613530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:16.613771 392 external_mini_cluster.cc:1428] Setting key 7b2ab6c45ed53a6c4840e6618a3259ba
I20260927 11:18:16.613929 3497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:16.615854 3497 catalog_manager.cc:1347] Generated new cluster ID: 18ced932d2ec42349e2743624f699a20
I20260927 11:18:16.615907 3497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:18:16 dist-test-slave-lsvg krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:16.625869 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:16.614533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.0.98.62:40741)
0927 11:18:16.614711 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:16.614721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:16.614752 (+ 31us) server_negotiation.cc:407] Connection header received
0927 11:18:16.614883 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:16.614892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:16.614954 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:16.615115 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:16.615892 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:16.617330 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:16.618394 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:16.619081 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:16.619203 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:16.622156 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:16.622161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:16.622161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:16.622175 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:16.624174 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:16.624582 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:16.624584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:16.624585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:16.624692 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:16.624899 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:16.624899 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:16.624899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:16.625240 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:16.625379 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:16.625497 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 11:18:16.625628 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260927 11:18:16.639318 3497 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:16.640101 3497 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:16.654839 3497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d204da5b3d044ce18cf772cf615facfd: Generated new TSK 0
I20260927 11:18:16.655073 3497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:18:16.680290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:16.762111 3511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:16.762346 3511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:16.762384 3511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:16.763561 3511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:16.763664 3511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:16.763710 3511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:16.765213 3511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:16.765473 3511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:16.765717 3511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:16.770725 3517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:16.770776 3516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:16.770877 3511 server_base.cc:1034] running on GCE node
W20260927 11:18:16.770764 3519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:16.771134 3511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:16.771334 3511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:16.772464 3511 hybrid_clock.cc:648] HybridClock initialized: now 1790507896772453 us; error 28 us; skew 500 ppm
Sep 27 11:18:16 dist-test-slave-lsvg krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:16.775097 3511 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:18:16.775475 3511 webserver.cc:460] Webserver started at http://127.0.98.1:35233/ using document root <none> and password file <none>
I20260927 11:18:16.775660 3511 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:16.775699 3511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:16.775789 3511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:16.776758 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "71e5513642cf4be680bda30dbd4cdb03"
format_stamp: "Formatted at 2026-09-27 11:18:16 on dist-test-slave-lsvg"
server_key: "bdec1b101adc7a09a2cc789900885964"
server_key_iv: "e002055834263916316d83868adf1a51"
server_key_version: "encryptionkey@0"
I20260927 11:18:16.777093 3511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "71e5513642cf4be680bda30dbd4cdb03"
format_stamp: "Formatted at 2026-09-27 11:18:16 on dist-test-slave-lsvg"
server_key: "bdec1b101adc7a09a2cc789900885964"
server_key_iv: "e002055834263916316d83868adf1a51"
server_key_version: "encryptionkey@0"
I20260927 11:18:16.778319 3511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:16.778961 3526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:16.779124 3511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:16.779237 3511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "71e5513642cf4be680bda30dbd4cdb03"
format_stamp: "Formatted at 2026-09-27 11:18:16 on dist-test-slave-lsvg"
server_key: "bdec1b101adc7a09a2cc789900885964"
server_key_iv: "e002055834263916316d83868adf1a51"
server_key_version: "encryptionkey@0"
I20260927 11:18:16.779310 3511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:16.790259 3511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:16.792775 3511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:16.792946 3511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:16.793252 3511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:16.793581 3511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:16.793634 3511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:16.793670 3511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:16.793696 3511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:16.799508 3511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42315
I20260927 11:18:16.799582 3639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42315 every 8 connection(s)
I20260927 11:18:16.799911 3511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:18:16.805374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 3511
I20260927 11:18:16.805451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:16.805634 392 external_mini_cluster.cc:1428] Setting key 97c6313a30f6502388e652b32aa2734e
Sep 27 11:18:16 dist-test-slave-lsvg krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:16.809677 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:16.800639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40741 (local address 127.0.98.1:51953)
0927 11:18:16.800812 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:16.800848 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:16.801405 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:16.801668 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:16.801672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:16.802211 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:16.802400 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:16.802405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:16.803118 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:16.803119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:16.803924 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:16.803926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:16.804354 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:16.804355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:16.804475 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:16.804918 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:16.804927 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:16.806538 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:16.808390 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:16.808396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:16.808397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:16.808612 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:16.808811 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:16.808812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:16.808812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:16.808902 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:16.809199 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:16.809204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:16.809322 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 11:18:16.809406 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 11:18:16.809521 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260927 11:18:16.809697 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:16.800663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51953 (local address 127.0.98.62:40741)
0927 11:18:16.800748 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:16.800750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:16.800931 (+ 181us) server_negotiation.cc:407] Connection header received
0927 11:18:16.801512 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:16.801513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:16.801524 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:16.801577 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:16.802501 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:16.803013 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:16.804028 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:16.804259 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:16.804344 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:16.806649 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:16.806651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:16.806651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:16.806659 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:16.808296 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:16.808682 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:16.808684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:16.808684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:16.808746 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:16.808964 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:16.808965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:16.808965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:16.809136 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:16.809212 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:16.809450 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 11:18:16.809548 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:18:16.810211 3640 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40741
I20260927 11:18:16.810326 3640 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:16.810505 3640 heartbeater.cc:511] Master 127.0.98.62:40741 requested a full tablet report, sending...
I20260927 11:18:16.811012 3449 ts_manager.cc:194] Registered new tserver with Master: 71e5513642cf4be680bda30dbd4cdb03 (127.0.98.1:42315)
I20260927 11:18:16.811600 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51953
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:18:16.831856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:16.910826 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:16.911027 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:16.911051 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:16.912210 3645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:16.912252 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:16.912278 3645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:16.913969 3645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:16.914211 3645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:16.914453 3645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:16.919544 3651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:16.919538 3650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:16.919538 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:16.919580 3645 server_base.cc:1034] running on GCE node
I20260927 11:18:16.919925 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:16.920167 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:16.921327 3645 hybrid_clock.cc:648] HybridClock initialized: now 1790507896921323 us; error 35 us; skew 500 ppm
Sep 27 11:18:16 dist-test-slave-lsvg krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:16.923871 3645 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:18:16.924288 3645 webserver.cc:460] Webserver started at http://127.0.98.2:33811/ using document root <none> and password file <none>
I20260927 11:18:16.924505 3645 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:16.924571 3645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:16.924723 3645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:16.925827 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "8b7e74129ab94df887a896da8520b3bb"
format_stamp: "Formatted at 2026-09-27 11:18:16 on dist-test-slave-lsvg"
server_key: "30b1c0d42837450178401d194530cd8d"
server_key_iv: "6f40c4ab29876a0a5dd97deb3ab0f1a1"
server_key_version: "encryptionkey@0"
I20260927 11:18:16.926222 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8b7e74129ab94df887a896da8520b3bb"
format_stamp: "Formatted at 2026-09-27 11:18:16 on dist-test-slave-lsvg"
server_key: "30b1c0d42837450178401d194530cd8d"
server_key_iv: "6f40c4ab29876a0a5dd97deb3ab0f1a1"
server_key_version: "encryptionkey@0"
I20260927 11:18:16.927493 3645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:18:16.928184 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:16.928344 3645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:16.928445 3645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "8b7e74129ab94df887a896da8520b3bb"
format_stamp: "Formatted at 2026-09-27 11:18:16 on dist-test-slave-lsvg"
server_key: "30b1c0d42837450178401d194530cd8d"
server_key_iv: "6f40c4ab29876a0a5dd97deb3ab0f1a1"
server_key_version: "encryptionkey@0"
I20260927 11:18:16.928517 3645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:16.962589 3645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:16.965003 3645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:16.965188 3645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:16.965435 3645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:16.965752 3645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:16.965807 3645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:16.965847 3645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:16.965875 3645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:16.971429 3645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35991
I20260927 11:18:16.971524 3773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35991 every 8 connection(s)
I20260927 11:18:16.971823 3645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:18:16.976600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 3645
I20260927 11:18:16.976716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:16.976955 392 external_mini_cluster.cc:1428] Setting key 1a9beafe021d6f2b526a37336f1ae7a7
Sep 27 11:18:16 dist-test-slave-lsvg krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:16.982672 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:16.972657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52303 (local address 127.0.98.62:40741)
0927 11:18:16.972729 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:16.972731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:16.972942 (+ 211us) server_negotiation.cc:407] Connection header received
0927 11:18:16.973490 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:16.973491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:16.973502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:16.973637 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:16.974569 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:16.975480 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:16.976424 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:16.976775 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:16.976844 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:16.979607 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:16.979608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:16.979609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:16.979617 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:16.981210 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:16.981674 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:16.981676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:16.981676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:16.981734 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:16.982006 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:16.982007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:16.982008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:16.982139 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:16.982213 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:16.982501 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 11:18:16.982582 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 11:18:16.982911 3776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:16.972639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40741 (local address 127.0.98.2:52303)
0927 11:18:16.972817 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:16.972853 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:16.973397 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:16.973689 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:16.973692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:16.974278 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:16.974466 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:16.974470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:16.975575 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:16.975576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:16.976349 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:16.976351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:16.976861 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:16.976862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:16.977010 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:16.977974 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:16.977982 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:16.979502 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:16.981298 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:16.981304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:16.981307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:16.981598 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:16.981809 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:16.981810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:16.981810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:16.981926 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:16.982204 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:16.982211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:16.982351 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 11:18:16.982461 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 11:18:16.982631 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260927 11:18:16.983328 3774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40741
I20260927 11:18:16.983443 3774 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:16.983631 3774 heartbeater.cc:511] Master 127.0.98.62:40741 requested a full tablet report, sending...
I20260927 11:18:16.984002 3449 ts_manager.cc:194] Registered new tserver with Master: 8b7e74129ab94df887a896da8520b3bb (127.0.98.2:35991)
I20260927 11:18:16.984388 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52303
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:18:17.005329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:17.084261 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:17.084481 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:17.084518 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:17.085680 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:17.085752 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:17.085796 3779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:17.087173 3779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:17.087426 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:17.087646 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:17.092609 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:17.092726 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:17.092851 3779 server_base.cc:1034] running on GCE node
W20260927 11:18:17.092928 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:17.093165 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:17.093401 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:17.094564 3779 hybrid_clock.cc:648] HybridClock initialized: now 1790507897094539 us; error 56 us; skew 500 ppm
Sep 27 11:18:17 dist-test-slave-lsvg krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:17.103412 3779 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:18:17.103876 3779 webserver.cc:460] Webserver started at http://127.0.98.3:41377/ using document root <none> and password file <none>
I20260927 11:18:17.104110 3779 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:17.104176 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:17.104331 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:17.105496 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "564e7ad1304b4f1f84fce01c7c41102c"
format_stamp: "Formatted at 2026-09-27 11:18:17 on dist-test-slave-lsvg"
server_key: "762e4b02a4a9640508d52644fd759e14"
server_key_iv: "79a75930bdc32effcb60b02dd6772d0b"
server_key_version: "encryptionkey@0"
I20260927 11:18:17.105849 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "564e7ad1304b4f1f84fce01c7c41102c"
format_stamp: "Formatted at 2026-09-27 11:18:17 on dist-test-slave-lsvg"
server_key: "762e4b02a4a9640508d52644fd759e14"
server_key_iv: "79a75930bdc32effcb60b02dd6772d0b"
server_key_version: "encryptionkey@0"
I20260927 11:18:17.107149 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:18:17.107846 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:17.108003 3779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:17.108098 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "564e7ad1304b4f1f84fce01c7c41102c"
format_stamp: "Formatted at 2026-09-27 11:18:17 on dist-test-slave-lsvg"
server_key: "762e4b02a4a9640508d52644fd759e14"
server_key_iv: "79a75930bdc32effcb60b02dd6772d0b"
server_key_version: "encryptionkey@0"
I20260927 11:18:17.108179 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:17.116006 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:17.117776 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:17.117955 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:17.118175 3779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:17.118471 3779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:17.118522 3779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:17.118564 3779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:17.118594 3779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:17.124241 3779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33735
I20260927 11:18:17.124277 3907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33735 every 8 connection(s)
I20260927 11:18:17.124572 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:17.129660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 3779
I20260927 11:18:17.129782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:17.129962 392 external_mini_cluster.cc:1428] Setting key 5c0461288e834e2f22ff0c6ed75fb43e
Sep 27 11:18:17 dist-test-slave-lsvg krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:17.133435 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:17.125296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43003 (local address 127.0.98.62:40741)
0927 11:18:17.125371 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:17.125373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:17.125529 (+ 156us) server_negotiation.cc:407] Connection header received
0927 11:18:17.125991 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:17.125993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:17.126007 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:17.126081 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:17.126939 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:17.127418 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:17.128347 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:17.128518 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:17.128565 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:17.130797 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:17.130798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:17.130799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:17.130807 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:17.132033 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:17.132441 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:17.132442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:17.132443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:17.132503 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:17.132752 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:17.132753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:17.132753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:17.132866 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:17.132941 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:17.133273 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 11:18:17.133348 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 11:18:17.133610 3910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:17.125261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40741 (local address 127.0.98.3:43003)
0927 11:18:17.125421 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:17.125453 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:17.125901 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:17.126142 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:17.126145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:17.126681 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:17.126864 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:17.126869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:17.127492 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:17.127493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:17.128278 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:17.128280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:17.128585 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:17.128586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:17.128676 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:17.129180 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:17.129188 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:17.130708 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:17.132134 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:17.132140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:17.132142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:17.132363 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:17.132569 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:17.132570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:17.132570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:17.132679 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:17.132932 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:17.132937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:17.133116 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 11:18:17.133238 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 11:18:17.133386 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260927 11:18:17.134007 3908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40741
I20260927 11:18:17.134112 3908 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:17.134251 3908 heartbeater.cc:511] Master 127.0.98.62:40741 requested a full tablet report, sending...
I20260927 11:18:17.134569 3449 ts_manager.cc:194] Registered new tserver with Master: 564e7ad1304b4f1f84fce01c7c41102c (127.0.98.3:33735)
I20260927 11:18:17.134918 3449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43003
I20260927 11:18:17.141573 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 11:18:17 dist-test-slave-lsvg krb5kdc[3404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507897, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 11:18:17 dist-test-slave-lsvg krb5kdc[3404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507897, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:17.162884 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:17.156354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60018 (local address 127.0.98.62:40741)
0927 11:18:17.156420 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:17.156422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:17.156525 (+ 103us) server_negotiation.cc:407] Connection header received
0927 11:18:17.156573 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:17.156573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:17.156581 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:17.156641 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:17.157230 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:17.157697 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:17.158419 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:17.158590 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:17.158638 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:17.160764 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:17.160765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:17.160765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:17.160771 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:17.161982 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:17.162272 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:17.162272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:17.162273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:17.162318 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:17.162467 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:17.162468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:17.162468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:17.162596 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:17.162668 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:17.162752 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 11:18:17.162819 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 11:18:17.163139 3449 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60018
I20260927 11:18:17.163586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 3511
I20260927 11:18:17.167654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 3645
I20260927 11:18:17.171741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 3779
I20260927 11:18:17.175709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 3417
2026-09-27T11:18:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1856 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:17 dist-test-slave-lsvg krb5kdc[3920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:17 dist-test-slave-lsvg krb5kdc[3920](info): set up 2 sockets
Sep 27 11:18:17 dist-test-slave-lsvg krb5kdc[3920](info): commencing operation
krb5kdc: starting...
W20260927 11:18:18.209020 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:18 dist-test-slave-lsvg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:18:18.328897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43941
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:18.408865 3933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:18.409135 3933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:18.409164 3933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:18.410296 3933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:18.410333 3933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:18.410347 3933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:18.410359 3933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:18.410372 3933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:18.411761 3933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:18.411949 3933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:18.412117 3933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:18.417449 3939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:18.417626 3933 server_base.cc:1034] running on GCE node
W20260927 11:18:18.417507 3941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:18.417742 3938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:18.418015 3933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:18.418279 3933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:18.419430 3933 hybrid_clock.cc:648] HybridClock initialized: now 1790507898419406 us; error 39 us; skew 500 ppm
Sep 27 11:18:18 dist-test-slave-lsvg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:18.422119 3933 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:18:18.422657 3933 webserver.cc:460] Webserver started at http://127.0.98.62:35959/ using document root <none> and password file <none>
I20260927 11:18:18.422891 3933 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:18.422959 3933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:18.423117 3933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:18.424257 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "8831df242a4c494e9b337e8cc5aca412"
format_stamp: "Formatted at 2026-09-27 11:18:18 on dist-test-slave-lsvg"
server_key: "303aedbf6fe1ca2db79563c600937659"
server_key_iv: "77f0d5931c448b5425ffb4ca676b8331"
server_key_version: "encryptionkey@0"
I20260927 11:18:18.424638 3933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "8831df242a4c494e9b337e8cc5aca412"
format_stamp: "Formatted at 2026-09-27 11:18:18 on dist-test-slave-lsvg"
server_key: "303aedbf6fe1ca2db79563c600937659"
server_key_iv: "77f0d5931c448b5425ffb4ca676b8331"
server_key_version: "encryptionkey@0"
I20260927 11:18:18.426090 3933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:18.426900 3948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:18.427114 3933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:18.427222 3933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "8831df242a4c494e9b337e8cc5aca412"
format_stamp: "Formatted at 2026-09-27 11:18:18 on dist-test-slave-lsvg"
server_key: "303aedbf6fe1ca2db79563c600937659"
server_key_iv: "77f0d5931c448b5425ffb4ca676b8331"
server_key_version: "encryptionkey@0"
I20260927 11:18:18.427299 3933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:18.465250 3933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:18.467739 3933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:18.467931 3933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:18.472055 3933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43941
I20260927 11:18:18.472101 4000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43941 every 8 connection(s)
I20260927 11:18:18.472503 3933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:18.473563 4001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:18.474232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 3933
I20260927 11:18:18.474351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:18.474526 392 external_mini_cluster.cc:1428] Setting key 1a10c79545cbe0079dbf49ec2ab95c73
I20260927 11:18:18.476771 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412: Bootstrap starting.
I20260927 11:18:18.477504 4001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:18.477807 4001 log.cc:824] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:18.478549 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412: No bootstrap required, opened a new log
I20260927 11:18:18.480542 4001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8831df242a4c494e9b337e8cc5aca412" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43941 } }
I20260927 11:18:18.480680 4001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:18.480702 4001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8831df242a4c494e9b337e8cc5aca412, State: Initialized, Role: FOLLOWER
I20260927 11:18:18.480839 4001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8831df242a4c494e9b337e8cc5aca412" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43941 } }
I20260927 11:18:18.480922 4001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:18.480945 4001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:18.481014 4001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:18.481581 4001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8831df242a4c494e9b337e8cc5aca412" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43941 } }
Sep 27 11:18:18 dist-test-slave-lsvg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:18.481700 4001 leader_election.cc:302] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8831df242a4c494e9b337e8cc5aca412; no voters:
I20260927 11:18:18.481881 4001 leader_election.cc:288] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:18.481930 4006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:18.482126 4006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [term 1 LEADER]: Becoming Leader. State: Replica: 8831df242a4c494e9b337e8cc5aca412, State: Running, Role: LEADER
I20260927 11:18:18.482139 4001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:18.482317 4006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8831df242a4c494e9b337e8cc5aca412" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43941 } }
I20260927 11:18:18.482873 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8831df242a4c494e9b337e8cc5aca412. Latest consensus state: current_term: 1 leader_uuid: "8831df242a4c494e9b337e8cc5aca412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8831df242a4c494e9b337e8cc5aca412" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43941 } } }
I20260927 11:18:18.483016 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:18.482793 4007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8831df242a4c494e9b337e8cc5aca412" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8831df242a4c494e9b337e8cc5aca412" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43941 } } }
I20260927 11:18:18.483242 4007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412 [sys.catalog]: This master's current role is: LEADER
W20260927 11:18:18.484005 4021 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:18:18.484047 4021 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:18:18.484144 4022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:18.484346 4022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:18.486289 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:18.475115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.0.98.62:43941)
0927 11:18:18.475288 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:18.475299 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:18.475338 (+ 39us) server_negotiation.cc:407] Connection header received
0927 11:18:18.475861 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:18.475870 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:18.475945 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:18.476116 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:18.476708 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:18.478128 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:18.478979 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:18.479419 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:18.479541 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:18.482230 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:18.482234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:18.482234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:18.482246 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:18.484477 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:18.484879 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:18.484882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:18.484882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:18.484989 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:18.485241 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:18.485242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:18.485242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:18.485536 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:18.485689 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:18.486029 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 11:18:18.486153 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260927 11:18:18.486425 4022 catalog_manager.cc:1347] Generated new cluster ID: b60c9c536ea34d579ff75926e54d8f77
I20260927 11:18:18.486459 4022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:18.498983 4022 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:18.499969 4022 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:18.504619 4022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8831df242a4c494e9b337e8cc5aca412: Generated new TSK 0
I20260927 11:18:18.504814 4022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:18:18.540498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43941
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:18.628785 4027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:18.629016 4027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:18.629040 4027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:18.630330 4027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:18.630375 4027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:18.630404 4027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:18.631997 4027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:18.632203 4027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:18.632396 4027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:18.637543 4035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:18.637560 4032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:18.637554 4033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:18.637733 4027 server_base.cc:1034] running on GCE node
I20260927 11:18:18.638029 4027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:18.638244 4027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:18.639384 4027 hybrid_clock.cc:648] HybridClock initialized: now 1790507898639370 us; error 28 us; skew 500 ppm
Sep 27 11:18:18 dist-test-slave-lsvg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:18.642040 4027 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:18:18.642519 4027 webserver.cc:460] Webserver started at http://127.0.98.1:43497/ using document root <none> and password file <none>
I20260927 11:18:18.642719 4027 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:18.642761 4027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:18.642863 4027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:18.643905 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "85f8a390d4b2439d8d5d54771866b202"
format_stamp: "Formatted at 2026-09-27 11:18:18 on dist-test-slave-lsvg"
server_key: "5f04400dbedc4b041c6615230176ff13"
server_key_iv: "f43317dd88b1a81ebb2e48d2bed689cd"
server_key_version: "encryptionkey@0"
I20260927 11:18:18.644215 4027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "85f8a390d4b2439d8d5d54771866b202"
format_stamp: "Formatted at 2026-09-27 11:18:18 on dist-test-slave-lsvg"
server_key: "5f04400dbedc4b041c6615230176ff13"
server_key_iv: "f43317dd88b1a81ebb2e48d2bed689cd"
server_key_version: "encryptionkey@0"
I20260927 11:18:18.645659 4027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:18.646523 4042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:18.646735 4027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:18.646819 4027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "85f8a390d4b2439d8d5d54771866b202"
format_stamp: "Formatted at 2026-09-27 11:18:18 on dist-test-slave-lsvg"
server_key: "5f04400dbedc4b041c6615230176ff13"
server_key_iv: "f43317dd88b1a81ebb2e48d2bed689cd"
server_key_version: "encryptionkey@0"
I20260927 11:18:18.646878 4027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:18.660606 4027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:18.663029 4027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:18.663146 4027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:18.663355 4027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:18.663681 4027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:18.663724 4027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:18.663748 4027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:18.663760 4027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:18.669533 4027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44231
I20260927 11:18:18.669625 4155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44231 every 8 connection(s)
I20260927 11:18:18.669950 4027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:18:18 dist-test-slave-lsvg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:18.676303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 4027
I20260927 11:18:18.676419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:18.676642 392 external_mini_cluster.cc:1428] Setting key 752e6a2794f6612e364c3f092b5cd539
I20260927 11:18:18.679826 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:18.670697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52903 (local address 127.0.98.62:43941)
0927 11:18:18.670763 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:18.670765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:18.671004 (+ 239us) server_negotiation.cc:407] Connection header received
0927 11:18:18.671943 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:18.671944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:18.671956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:18.672009 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:18.672603 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:18.673220 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:18.674281 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:18.674506 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:18.674571 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:18.676847 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:18.676849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:18.676849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:18.676858 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:18.678393 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:18.678845 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:18.678846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:18.678847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:18.678909 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:18.679163 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:18.679164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:18.679165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:18.679285 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:18.679359 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:18.679648 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 11:18:18.679732 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 11:18:18.680050 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:18.670663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43941 (local address 127.0.98.1:52903)
0927 11:18:18.670847 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:18.670887 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:18.671843 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:18.672080 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:18.672083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:18.672324 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:18.672512 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:18.672516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:18.673296 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:18.673297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:18.674186 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:18.674188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:18.674589 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:18.674590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:18.674688 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:18.675225 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:18.675233 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:18.676741 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:18.678506 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:18.678513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:18.678515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:18.678752 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:18.678978 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:18.678979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:18.678979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:18.679096 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:18.679351 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:18.679357 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:18.679513 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 11:18:18.679627 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 11:18:18.679792 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260927 11:18:18.680529 4156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43941
I20260927 11:18:18.680649 4156 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:18.680828 4156 heartbeater.cc:511] Master 127.0.98.62:43941 requested a full tablet report, sending...
I20260927 11:18:18.681427 3965 ts_manager.cc:194] Registered new tserver with Master: 85f8a390d4b2439d8d5d54771866b202 (127.0.98.1:44231)
I20260927 11:18:18.682423 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52903
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:18:18.705096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43941
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:18.786465 4161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:18.786736 4161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:18.786763 4161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:18.787918 4161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:18.787956 4161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:18.787982 4161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:18.789522 4161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:18.789716 4161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:18.789896 4161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:18.795156 4166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:18.795317 4161 server_base.cc:1034] running on GCE node
W20260927 11:18:18.795188 4169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:18.795184 4167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:18.795842 4161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:18.796166 4161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:18.797345 4161 hybrid_clock.cc:648] HybridClock initialized: now 1790507898797324 us; error 36 us; skew 500 ppm
Sep 27 11:18:18 dist-test-slave-lsvg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:18.799971 4161 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:18:18.800465 4161 webserver.cc:460] Webserver started at http://127.0.98.2:46019/ using document root <none> and password file <none>
I20260927 11:18:18.800699 4161 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:18.800765 4161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:18.800915 4161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:18.802071 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "cafb88fbed5a411498c9fffd003d98d1"
format_stamp: "Formatted at 2026-09-27 11:18:18 on dist-test-slave-lsvg"
server_key: "1cc20c95b894c31f282246306c01e080"
server_key_iv: "407cf6e5a83bf06c9f9f98fe6c39f8a8"
server_key_version: "encryptionkey@0"
I20260927 11:18:18.802448 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cafb88fbed5a411498c9fffd003d98d1"
format_stamp: "Formatted at 2026-09-27 11:18:18 on dist-test-slave-lsvg"
server_key: "1cc20c95b894c31f282246306c01e080"
server_key_iv: "407cf6e5a83bf06c9f9f98fe6c39f8a8"
server_key_version: "encryptionkey@0"
I20260927 11:18:18.803840 4161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:18:18.804622 4176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:18.804791 4161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:18.804898 4161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "cafb88fbed5a411498c9fffd003d98d1"
format_stamp: "Formatted at 2026-09-27 11:18:18 on dist-test-slave-lsvg"
server_key: "1cc20c95b894c31f282246306c01e080"
server_key_iv: "407cf6e5a83bf06c9f9f98fe6c39f8a8"
server_key_version: "encryptionkey@0"
I20260927 11:18:18.804971 4161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:18.833505 4161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:18.835692 4161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:18.835858 4161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:18.836107 4161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:18.836436 4161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:18.836489 4161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:18.836530 4161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:18.836566 4161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:18.842514 4161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35909
I20260927 11:18:18.842576 4289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35909 every 8 connection(s)
I20260927 11:18:18.842891 4161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:18:18 dist-test-slave-lsvg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:18.850224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 4161
I20260927 11:18:18.850343 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:18.850591 392 external_mini_cluster.cc:1428] Setting key 36e826bf92bee93502086c1a462bcaaa
I20260927 11:18:18.852885 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:18.843639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48321 (local address 127.0.98.62:43941)
0927 11:18:18.843739 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:18.843741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:18.843903 (+ 162us) server_negotiation.cc:407] Connection header received
0927 11:18:18.844830 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:18.844831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:18.844841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:18.844902 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:18.845588 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:18.846076 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:18.847115 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:18.847299 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:18.847354 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:18.849637 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:18.849639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:18.849639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:18.849647 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:18.851157 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:18.851692 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:18.851693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:18.851693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:18.851742 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:18.852053 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:18.852055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:18.852055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:18.852191 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:18.852271 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:18.852646 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 11:18:18.852757 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 11:18:18.853228 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:18.843571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43941 (local address 127.0.98.2:48321)
0927 11:18:18.843750 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:18.843793 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:18.844732 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:18.844979 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:18.844983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:18.845271 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:18.845500 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:18.845505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:18.846185 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:18.846186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:18.847025 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:18.847027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:18.847372 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:18.847373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:18.847509 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:18.848083 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:18.848095 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:18.849507 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:18.851251 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:18.851263 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:18.851264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:18.851620 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:18.851807 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:18.851807 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:18.851808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:18.851964 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:18.852281 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:18.852287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:18.852486 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 11:18:18.852606 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 11:18:18.852790 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260927 11:18:18.853705 4290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43941
I20260927 11:18:18.853802 4290 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:18.854003 4290 heartbeater.cc:511] Master 127.0.98.62:43941 requested a full tablet report, sending...
I20260927 11:18:18.854398 3965 ts_manager.cc:194] Registered new tserver with Master: cafb88fbed5a411498c9fffd003d98d1 (127.0.98.2:35909)
I20260927 11:18:18.854928 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48321
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:18:18.876777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43941
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:18.958429 4295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:18.958647 4295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:18.958669 4295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:18.959774 4295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:18.959813 4295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:18.959839 4295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:18.961309 4295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:18.961511 4295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:18.961690 4295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:18.966753 4300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:18.966756 4303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:18.966749 4301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:18.966789 4295 server_base.cc:1034] running on GCE node
I20260927 11:18:18.967312 4295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:18.967582 4295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:18.968748 4295 hybrid_clock.cc:648] HybridClock initialized: now 1790507898968737 us; error 27 us; skew 500 ppm
Sep 27 11:18:18 dist-test-slave-lsvg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:18.971365 4295 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:18:18.971843 4295 webserver.cc:460] Webserver started at http://127.0.98.3:45757/ using document root <none> and password file <none>
I20260927 11:18:18.972085 4295 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:18.972152 4295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:18.972306 4295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:18.973444 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "8e6268540bf748a79aed758c14a93527"
format_stamp: "Formatted at 2026-09-27 11:18:18 on dist-test-slave-lsvg"
server_key: "ac27ca79ed7b92d3a193feb355123348"
server_key_iv: "4584cc170e655dcbedd4065823bb4f3c"
server_key_version: "encryptionkey@0"
I20260927 11:18:18.973795 4295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8e6268540bf748a79aed758c14a93527"
format_stamp: "Formatted at 2026-09-27 11:18:18 on dist-test-slave-lsvg"
server_key: "ac27ca79ed7b92d3a193feb355123348"
server_key_iv: "4584cc170e655dcbedd4065823bb4f3c"
server_key_version: "encryptionkey@0"
I20260927 11:18:18.975097 4295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:18.975809 4310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:18.975971 4295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:18.976073 4295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "8e6268540bf748a79aed758c14a93527"
format_stamp: "Formatted at 2026-09-27 11:18:18 on dist-test-slave-lsvg"
server_key: "ac27ca79ed7b92d3a193feb355123348"
server_key_iv: "4584cc170e655dcbedd4065823bb4f3c"
server_key_version: "encryptionkey@0"
I20260927 11:18:18.976145 4295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:19.007323 4295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:19.009865 4295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:19.010027 4295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:19.010263 4295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:19.010592 4295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:19.010645 4295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:19.010689 4295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:19.010715 4295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:19.016485 4295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35411
I20260927 11:18:19.016528 4423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35411 every 8 connection(s)
I20260927 11:18:19.016839 4295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:19.021824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 4295
I20260927 11:18:19.021932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:19.022125 392 external_mini_cluster.cc:1428] Setting key 860de053c751b8f98bb9d4997f381962
Sep 27 11:18:19 dist-test-slave-lsvg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:19.026006 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:19.017511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48357 (local address 127.0.98.62:43941)
0927 11:18:19.017574 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:19.017576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:19.017788 (+ 212us) server_negotiation.cc:407] Connection header received
0927 11:18:19.018724 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:19.018725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:19.018736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:19.018789 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:19.019379 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:19.019858 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:19.020823 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:19.020998 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:19.021092 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:19.023524 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:19.023525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:19.023525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:19.023531 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:19.024755 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:19.025233 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:19.025234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:19.025234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:19.025284 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:19.025494 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:19.025494 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:19.025495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:19.025608 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:19.025657 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:19.025877 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 11:18:19.025941 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:18:19.026190 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:19.017502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43941 (local address 127.0.98.3:48357)
0927 11:18:19.017672 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:19.017708 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:19.018623 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:19.018861 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:19.018865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:19.019126 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:19.019314 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:19.019318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:19.019929 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:19.019930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:19.020749 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:19.020752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:19.021119 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:19.021120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:19.021240 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:19.021785 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:19.021794 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:19.023452 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:19.024848 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:19.024854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:19.024855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:19.025169 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:19.025343 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:19.025344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:19.025344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:19.025442 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:19.025661 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:19.025665 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:19.025774 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 11:18:19.025856 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 11:18:19.025971 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260927 11:18:19.026579 4424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43941
I20260927 11:18:19.026695 4424 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:19.026866 4424 heartbeater.cc:511] Master 127.0.98.62:43941 requested a full tablet report, sending...
I20260927 11:18:19.027217 3965 ts_manager.cc:194] Registered new tserver with Master: 8e6268540bf748a79aed758c14a93527 (127.0.98.3:35411)
I20260927 11:18:19.027643 3965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48357
I20260927 11:18:19.033921 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 11:18:19 dist-test-slave-lsvg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 11:18:19 dist-test-slave-lsvg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:18:19.056738 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:19.048913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.0.98.1:44231)
0927 11:18:19.049164 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:19.049169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:19.049179 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:19.049204 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:19.049206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:19.049219 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:19.049284 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:19.049863 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:19.050686 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:19.051419 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:19.051758 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:19.051818 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:19.054083 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:19.054100 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:19.054100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:19.054107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:19.055761 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:19.056065 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:19.056068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:19.056069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:19.056133 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:19.056279 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:19.056280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:19.056280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:19.056404 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:19.056533 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:19.056593 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 11:18:19.056656 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
Sep 27 11:18:19 dist-test-slave-lsvg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:19.064821 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:19.058111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.0.98.62:43941)
0927 11:18:19.058173 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:19.058175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:19.058184 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:19.058244 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:19.058245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:19.058254 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:19.058303 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:19.058792 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:19.059294 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:19.059967 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:19.060167 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:19.060227 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:19.062482 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:19.062484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:19.062484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:19.062491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:19.063890 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:19.064181 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:19.064182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:19.064182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:19.064231 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:19.064370 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:19.064370 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:19.064370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:19.064502 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:19.064571 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:19.064659 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 11:18:19.064714 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 11:18:19.065131 3965 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54206
I20260927 11:18:19.065492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 4027
I20260927 11:18:19.072171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 4161
I20260927 11:18:19.077680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 4295
I20260927 11:18:19.082113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 3933
2026-09-27T11:18:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1907 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:19 dist-test-slave-lsvg krb5kdc[4436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:19 dist-test-slave-lsvg krb5kdc[4436](info): set up 2 sockets
Sep 27 11:18:19 dist-test-slave-lsvg krb5kdc[4436](info): commencing operation
krb5kdc: starting...
W20260927 11:18:20.157686 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:20 dist-test-slave-lsvg krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:18:20.280390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46193
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:20.360587 4449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:20.360846 4449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:20.360885 4449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:20.362057 4449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:20.362129 4449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:20.362152 4449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:20.362174 4449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:20.362203 4449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:20.363684 4449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:20.363967 4449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:20.364197 4449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:20.369133 4455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:20.369331 4454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:20.369407 4449 server_base.cc:1034] running on GCE node
W20260927 11:18:20.369454 4457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:20.369788 4449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:20.370050 4449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:20.371188 4449 hybrid_clock.cc:648] HybridClock initialized: now 1790507900371150 us; error 60 us; skew 500 ppm
Sep 27 11:18:20 dist-test-slave-lsvg krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:20.374116 4449 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:18:20.374569 4449 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:20.374625 4449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:20.374811 4449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:20.375993 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "84dbcef456b846ec9c64a0aa0fdbabbf"
format_stamp: "Formatted at 2026-09-27 11:18:20 on dist-test-slave-lsvg"
server_key: "e9eb6ee851998a5d7b57fb4bfa914ad6"
server_key_iv: "2e57587d5dd0b1d34eb65e9d0f6a3f84"
server_key_version: "encryptionkey@0"
I20260927 11:18:20.376389 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "84dbcef456b846ec9c64a0aa0fdbabbf"
format_stamp: "Formatted at 2026-09-27 11:18:20 on dist-test-slave-lsvg"
server_key: "e9eb6ee851998a5d7b57fb4bfa914ad6"
server_key_iv: "2e57587d5dd0b1d34eb65e9d0f6a3f84"
server_key_version: "encryptionkey@0"
I20260927 11:18:20.377902 4449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:20.378757 4463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:20.378954 4449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:18:20.379076 4449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "84dbcef456b846ec9c64a0aa0fdbabbf"
format_stamp: "Formatted at 2026-09-27 11:18:20 on dist-test-slave-lsvg"
server_key: "e9eb6ee851998a5d7b57fb4bfa914ad6"
server_key_iv: "2e57587d5dd0b1d34eb65e9d0f6a3f84"
server_key_version: "encryptionkey@0"
I20260927 11:18:20.379158 4449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:20.404177 4449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:20.406375 4449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:20.406555 4449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:20.410930 4449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46193
I20260927 11:18:20.410955 4515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46193 every 8 connection(s)
I20260927 11:18:20.411271 4449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:20.412232 4516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:20.414619 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf: Bootstrap starting.
I20260927 11:18:20.415246 4516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:20.415402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 4449
I20260927 11:18:20.415513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:20.415609 4516 log.cc:824] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:20.415714 392 external_mini_cluster.cc:1428] Setting key c3c144c27bb3a077517dd161d0bb60fc
I20260927 11:18:20.416388 4516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf: No bootstrap required, opened a new log
I20260927 11:18:20.418556 4516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84dbcef456b846ec9c64a0aa0fdbabbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } }
I20260927 11:18:20.418689 4516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:20.418710 4516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84dbcef456b846ec9c64a0aa0fdbabbf, State: Initialized, Role: FOLLOWER
I20260927 11:18:20.418839 4516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84dbcef456b846ec9c64a0aa0fdbabbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } }
I20260927 11:18:20.418920 4516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:20.418985 4516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:20.419031 4516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:20.419551 4516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84dbcef456b846ec9c64a0aa0fdbabbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } }
I20260927 11:18:20.419667 4516 leader_election.cc:302] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84dbcef456b846ec9c64a0aa0fdbabbf; no voters:
I20260927 11:18:20.419831 4516 leader_election.cc:288] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:20.419886 4521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:20.420022 4521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [term 1 LEADER]: Becoming Leader. State: Replica: 84dbcef456b846ec9c64a0aa0fdbabbf, State: Running, Role: LEADER
I20260927 11:18:20.420091 4516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:20.420105 4521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84dbcef456b846ec9c64a0aa0fdbabbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } }
I20260927 11:18:20.420460 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84dbcef456b846ec9c64a0aa0fdbabbf. Latest consensus state: current_term: 1 leader_uuid: "84dbcef456b846ec9c64a0aa0fdbabbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84dbcef456b846ec9c64a0aa0fdbabbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } } }
I20260927 11:18:20.420565 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:20.420694 4523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84dbcef456b846ec9c64a0aa0fdbabbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84dbcef456b846ec9c64a0aa0fdbabbf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } } }
I20260927 11:18:20.420782 4523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:20.421169 4530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:20.421680 4530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 11:18:20 dist-test-slave-lsvg krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:20.423650 4530 catalog_manager.cc:1347] Generated new cluster ID: 7d6dca560fee49959699feae023b57e7
I20260927 11:18:20.423697 4530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:20.432034 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:20.416218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.0.98.62:46193)
0927 11:18:20.416411 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:20.416423 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:20.416477 (+ 54us) server_negotiation.cc:407] Connection header received
0927 11:18:20.416915 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:20.416922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:20.416966 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:20.417158 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:20.417800 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.418667 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.419580 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.420243 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.420393 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:20.423315 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:20.423319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:20.423319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:20.423331 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:20.430419 (+ 7088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:20.430852 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:20.430856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:20.430857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:20.430955 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:20.431162 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:20.431163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:20.431164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:20.431412 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:20.431582 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:20.431673 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 11:18:20.431812 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260927 11:18:20.469031 4530 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:20.469641 4530 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:20.472577 4530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf: Generated new TSK 0
I20260927 11:18:20.472810 4530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:18:20.513154 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46193
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:20.599409 4542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:20.599622 4542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:20.599647 4542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:20.600912 4542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:20.600957 4542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:20.600986 4542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:20.602955 4542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:20.603215 4542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:20.603474 4542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:20.608979 4550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:20.608980 4547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:20.608980 4548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:20.609383 4542 server_base.cc:1034] running on GCE node
I20260927 11:18:20.609593 4542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:20.609809 4542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:20.610952 4542 hybrid_clock.cc:648] HybridClock initialized: now 1790507900610940 us; error 29 us; skew 500 ppm
Sep 27 11:18:20 dist-test-slave-lsvg krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:20.613926 4542 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:18:20.614312 4542 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:20.614387 4542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:20.614538 4542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:20.615646 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "8e4d544e3cf54b03b4520af7707fd351"
format_stamp: "Formatted at 2026-09-27 11:18:20 on dist-test-slave-lsvg"
server_key: "2f8ac29d24c5de8dcc987b881a4cd00b"
server_key_iv: "61f99dd3858dc8b6cf2173a0500ddd8b"
server_key_version: "encryptionkey@0"
I20260927 11:18:20.616017 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8e4d544e3cf54b03b4520af7707fd351"
format_stamp: "Formatted at 2026-09-27 11:18:20 on dist-test-slave-lsvg"
server_key: "2f8ac29d24c5de8dcc987b881a4cd00b"
server_key_iv: "61f99dd3858dc8b6cf2173a0500ddd8b"
server_key_version: "encryptionkey@0"
I20260927 11:18:20.617484 4542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:20.618386 4556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:20.618631 4542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:20.618757 4542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "8e4d544e3cf54b03b4520af7707fd351"
format_stamp: "Formatted at 2026-09-27 11:18:20 on dist-test-slave-lsvg"
server_key: "2f8ac29d24c5de8dcc987b881a4cd00b"
server_key_iv: "61f99dd3858dc8b6cf2173a0500ddd8b"
server_key_version: "encryptionkey@0"
I20260927 11:18:20.618849 4542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:20.631613 4542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:20.634248 4542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:20.634439 4542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:20.634696 4542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:20.635038 4542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:20.635092 4542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:20.635128 4542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:20.635154 4542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:20.641156 4542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34361
I20260927 11:18:20.641212 4669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34361 every 8 connection(s)
I20260927 11:18:20.641539 4542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:18:20 dist-test-slave-lsvg krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:20.648665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 4542
I20260927 11:18:20.648808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:20.649092 392 external_mini_cluster.cc:1428] Setting key 05a0e8b70eeff4a7e6b251a23066fa21
I20260927 11:18:20.652411 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:20.642374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40353 (local address 127.0.98.62:46193)
0927 11:18:20.642457 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:20.642459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:20.642661 (+ 202us) server_negotiation.cc:407] Connection header received
0927 11:18:20.643243 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:20.643244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:20.643264 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:20.643309 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:20.644406 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.644978 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.646140 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.646371 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.646451 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:20.648995 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:20.648997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:20.648997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:20.649005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:20.650736 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:20.651376 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:20.651377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:20.651378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:20.651434 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:20.651736 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:20.651736 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:20.651737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:20.651851 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:20.651903 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:20.652265 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 11:18:20.652344 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 11:18:20.652771 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:20.642334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46193 (local address 127.0.98.1:40353)
0927 11:18:20.642519 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:20.642567 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:20.643113 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:20.643410 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:20.643419 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:20.643724 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:20.644287 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:20.644293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.645112 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.645113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:20.646033 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:20.646036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.646472 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.646473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:20.646613 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:20.647179 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:20.647188 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:20.648885 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:20.650867 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:20.650875 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:20.650878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:20.651258 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:20.651507 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:20.651509 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:20.651509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:20.651667 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:20.651915 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:20.651921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:20.652112 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 11:18:20.652248 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 11:18:20.652448 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260927 11:18:20.653443 4670 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46193
I20260927 11:18:20.653568 4670 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:20.653766 4670 heartbeater.cc:511] Master 127.0.98.62:46193 requested a full tablet report, sending...
I20260927 11:18:20.654498 4480 ts_manager.cc:194] Registered new tserver with Master: 8e4d544e3cf54b03b4520af7707fd351 (127.0.98.1:34361)
I20260927 11:18:20.655591 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40353
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:18:20.677039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46193
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:20.759683 4675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:20.759909 4675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:20.759934 4675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:20.761025 4675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:20.761085 4675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:20.761123 4675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:20.762549 4675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:20.762730 4675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:20.762912 4675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:20.767905 4683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:20.767915 4680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:20.768003 4681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:20.768122 4675 server_base.cc:1034] running on GCE node
I20260927 11:18:20.768440 4675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:20.768652 4675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:20.769804 4675 hybrid_clock.cc:648] HybridClock initialized: now 1790507900769784 us; error 45 us; skew 500 ppm
Sep 27 11:18:20 dist-test-slave-lsvg krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:20.772330 4675 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:18:20.772697 4675 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:20.772746 4675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:20.772843 4675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:20.773924 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "ac12926cd10f4a9b92b419bc4dfbd140"
format_stamp: "Formatted at 2026-09-27 11:18:20 on dist-test-slave-lsvg"
server_key: "979a84a92e3d4af72e9b56c1aadde007"
server_key_iv: "a394a5d3f9d40bed5a7b862cb703d829"
server_key_version: "encryptionkey@0"
I20260927 11:18:20.774241 4675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ac12926cd10f4a9b92b419bc4dfbd140"
format_stamp: "Formatted at 2026-09-27 11:18:20 on dist-test-slave-lsvg"
server_key: "979a84a92e3d4af72e9b56c1aadde007"
server_key_iv: "a394a5d3f9d40bed5a7b862cb703d829"
server_key_version: "encryptionkey@0"
I20260927 11:18:20.775517 4675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:18:20.776273 4689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:20.776526 4675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:20.776610 4675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "ac12926cd10f4a9b92b419bc4dfbd140"
format_stamp: "Formatted at 2026-09-27 11:18:20 on dist-test-slave-lsvg"
server_key: "979a84a92e3d4af72e9b56c1aadde007"
server_key_iv: "a394a5d3f9d40bed5a7b862cb703d829"
server_key_version: "encryptionkey@0"
I20260927 11:18:20.776710 4675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:20.793385 4675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:20.795542 4675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:20.795693 4675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:20.795948 4675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:20.796304 4675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:20.796360 4675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:20.796403 4675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:20.796430 4675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:20.802541 4675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45139
I20260927 11:18:20.802578 4802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45139 every 8 connection(s)
I20260927 11:18:20.802888 4675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:18:20 dist-test-slave-lsvg krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:20.812274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 4675
I20260927 11:18:20.812397 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:20.812506 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:20.803699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51703 (local address 127.0.98.62:46193)
0927 11:18:20.803798 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:20.803800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:20.803965 (+ 165us) server_negotiation.cc:407] Connection header received
0927 11:18:20.804464 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:20.804465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:20.804476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:20.804569 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:20.805530 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.806039 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.807080 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.807270 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.807338 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:20.809713 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:20.809715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:20.809715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:20.809722 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:20.811170 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:20.811580 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:20.811580 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:20.811581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:20.811628 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:20.811892 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:20.811893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:20.811893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:20.811992 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:20.812041 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:20.812322 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 11:18:20.812411 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 11:18:20.812618 392 external_mini_cluster.cc:1428] Setting key bdb0ae83041760dd04b17ceb80f7ca2d
I20260927 11:18:20.812753 4805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:20.803663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46193 (local address 127.0.98.2:51703)
0927 11:18:20.803826 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:20.803867 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:20.804343 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:20.804597 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:20.804600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:20.804861 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:20.805433 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:20.805439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.806148 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.806149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:20.806988 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:20.806991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.807378 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.807379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:20.807496 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:20.807993 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:20.808002 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:20.809621 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:20.811255 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:20.811262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:20.811264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:20.811509 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:20.811709 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:20.811710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:20.811711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:20.811827 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:20.812049 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:20.812053 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:20.812171 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 11:18:20.812266 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 11:18:20.812467 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260927 11:18:20.813258 4803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46193
I20260927 11:18:20.813370 4803 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:20.813531 4803 heartbeater.cc:511] Master 127.0.98.62:46193 requested a full tablet report, sending...
I20260927 11:18:20.813934 4480 ts_manager.cc:194] Registered new tserver with Master: ac12926cd10f4a9b92b419bc4dfbd140 (127.0.98.2:45139)
I20260927 11:18:20.814333 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51703
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:18:20.840569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46193
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:20.936282 4808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:20.936492 4808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:20.936517 4808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:20.937793 4808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:20.937839 4808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:20.937866 4808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:20.939368 4808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:20.939621 4808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:20.939879 4808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:20.944836 4813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:20.944828 4816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:20.944828 4814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:20.945241 4808 server_base.cc:1034] running on GCE node
I20260927 11:18:20.945438 4808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:20.945636 4808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:20.946779 4808 hybrid_clock.cc:648] HybridClock initialized: now 1790507900946766 us; error 28 us; skew 500 ppm
Sep 27 11:18:20 dist-test-slave-lsvg krb5kdc[4436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:20.949370 4808 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:18:20.949776 4808 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:20.949851 4808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:20.950008 4808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:20.951081 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "6233c7f3a16f43b9a818eee7e224a3aa"
format_stamp: "Formatted at 2026-09-27 11:18:20 on dist-test-slave-lsvg"
server_key: "e49847682bfbb62d4d26c0a26040df29"
server_key_iv: "eac1f69a3b2b2550a93e444f27d1c569"
server_key_version: "encryptionkey@0"
I20260927 11:18:20.951437 4808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6233c7f3a16f43b9a818eee7e224a3aa"
format_stamp: "Formatted at 2026-09-27 11:18:20 on dist-test-slave-lsvg"
server_key: "e49847682bfbb62d4d26c0a26040df29"
server_key_iv: "eac1f69a3b2b2550a93e444f27d1c569"
server_key_version: "encryptionkey@0"
I20260927 11:18:20.952728 4808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 11:18:20.953559 4822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:20.953727 4808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:20.953830 4808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "6233c7f3a16f43b9a818eee7e224a3aa"
format_stamp: "Formatted at 2026-09-27 11:18:20 on dist-test-slave-lsvg"
server_key: "e49847682bfbb62d4d26c0a26040df29"
server_key_iv: "eac1f69a3b2b2550a93e444f27d1c569"
server_key_version: "encryptionkey@0"
I20260927 11:18:20.953920 4808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:20.974536 4808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:20.977015 4808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:20.977239 4808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:20.977468 4808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:20.977785 4808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:20.977826 4808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:20.977870 4808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:20.977896 4808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:20.984218 4808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39215
I20260927 11:18:20.984253 4935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39215 every 8 connection(s)
I20260927 11:18:20.984586 4808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:20.985754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 4808
I20260927 11:18:20.985870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:20.986050 392 external_mini_cluster.cc:1428] Setting key ceb26d4201d19c07670cea884a6af503
Sep 27 11:18:20 dist-test-slave-lsvg krb5kdc[4436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:20.994421 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:20.985365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38061 (local address 127.0.98.62:46193)
0927 11:18:20.985451 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:20.985453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:20.985649 (+ 196us) server_negotiation.cc:407] Connection header received
0927 11:18:20.986217 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:20.986218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:20.986230 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:20.986325 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:20.987298 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.987825 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.988906 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.989191 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.989282 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:20.991437 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:20.991438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:20.991439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:20.991447 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:20.992932 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:20.993412 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:20.993413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:20.993414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:20.993475 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:20.993738 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:20.993739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:20.993739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:20.993857 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:20.993932 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:20.994238 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 11:18:20.994328 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 11:18:20.994621 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:20.985345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46193 (local address 127.0.98.3:38061)
0927 11:18:20.985511 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:20.985571 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:20.986083 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:20.986412 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:20.986417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:20.986681 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:20.987181 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:20.987186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.987933 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.987934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:20.988819 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:20.988821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:20.989282 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:20.989283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:20.989418 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:20.989982 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:20.989994 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:20.991336 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:20.993033 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:20.993040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:20.993042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:20.993295 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:20.993543 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:20.993544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:20.993544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:20.993665 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:20.993922 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:20.993928 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:20.994080 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 11:18:20.994206 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 11:18:20.994369 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260927 11:18:20.995119 4936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46193
I20260927 11:18:20.995395 4936 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:20.995642 4936 heartbeater.cc:511] Master 127.0.98.62:46193 requested a full tablet report, sending...
I20260927 11:18:20.996045 4480 ts_manager.cc:194] Registered new tserver with Master: 6233c7f3a16f43b9a818eee7e224a3aa (127.0.98.3:39215)
I20260927 11:18:20.996380 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38061
I20260927 11:18:20.997866 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:18:21.009361 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.001313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.0.98.62:46193)
0927 11:18:21.001357 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.001359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.001678 (+ 319us) server_negotiation.cc:407] Connection header received
0927 11:18:21.001710 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.001711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.001722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.001784 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:21.002494 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.003167 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.004286 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.004584 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.004658 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:21.006384 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:21.006386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:21.006386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:21.006393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:21.008079 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:21.008465 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:21.008466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:21.008467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:21.008530 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:21.008736 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:21.008737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:21.008738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:21.008901 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:21.009020 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.009164 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.009252 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:18:21.010983 4480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:21.012663 4480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:21.019646 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.015499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.0.98.1:34361)
0927 11:18:21.015756 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.015759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.015770 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:18:21.015924 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.015926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.015943 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.016018 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:21.016280 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.017448 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.018720 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.019230 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.019323 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:21.019387 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.019509 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.019562 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":76,"threads_started":1}
I20260927 11:18:21.019637 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.015369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34361 (local address 127.0.0.1:54826)
0927 11:18:21.015672 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.015716 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:21.015789 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:21.016121 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:21.016125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:21.016127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:21.016198 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:21.016200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.017555 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.017555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:21.018632 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:21.018634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.019322 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.019323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:21.019401 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:21.019415 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 11:18:21.019493 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 11:18:21.019549 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":60,"threads_started":1}
I20260927 11:18:21.019775 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.015369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45139 (local address 127.0.0.1:38624)
0927 11:18:21.015816 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.015825 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:21.015900 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:21.016225 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:21.016226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:21.016227 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:21.016307 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:21.016308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.017285 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.017286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:21.018603 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:21.018606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.019525 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.019525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:21.019582 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:21.019585 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:21.019635 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 11:18:21.019693 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":18,"thread_start_us":91,"threads_started":1}
I20260927 11:18:21.019863 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.015704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.0.98.3:39215)
0927 11:18:21.015873 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.015877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.016147 (+ 270us) server_negotiation.cc:407] Connection header received
0927 11:18:21.016191 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.016193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.016207 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.016286 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:21.016392 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.017468 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.018575 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.019320 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.019395 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:21.019470 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.019737 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.019777 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20260927 11:18:21.020057 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.015499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.0.98.2:45139)
0927 11:18:21.015679 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.015684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.015876 (+ 192us) server_negotiation.cc:407] Connection header received
0927 11:18:21.016031 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.016035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.016053 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.016130 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:21.016412 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.017188 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.018691 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.019365 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.019444 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:21.019496 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.019875 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.019935 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
I20260927 11:18:21.020216 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.015605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39215 (local address 127.0.0.1:47590)
0927 11:18:21.016031 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.016041 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:21.016083 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:21.016294 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:21.016295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:21.016297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:21.016333 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:21.016335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.017572 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.017573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:21.018489 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:21.018492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.019508 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.019509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:21.019581 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:21.019587 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 11:18:21.019635 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 11:18:21.019692 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":7,"thread_start_us":84,"threads_started":1}
I20260927 11:18:21.021039 4870 tablet_service.cc:1467] Processing CreateTablet for tablet 54a79bccbffb453e8a5a002cb5917b4e (DEFAULT_TABLE table=test-table [id=99d69f64d0944d208171f493388c37af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:21.021046 4737 tablet_service.cc:1467] Processing CreateTablet for tablet 54a79bccbffb453e8a5a002cb5917b4e (DEFAULT_TABLE table=test-table [id=99d69f64d0944d208171f493388c37af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:21.021155 4604 tablet_service.cc:1467] Processing CreateTablet for tablet 54a79bccbffb453e8a5a002cb5917b4e (DEFAULT_TABLE table=test-table [id=99d69f64d0944d208171f493388c37af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:21.021431 4870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a79bccbffb453e8a5a002cb5917b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:21.021437 4604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a79bccbffb453e8a5a002cb5917b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:21.021436 4737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a79bccbffb453e8a5a002cb5917b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:21.024119 4953 tablet_bootstrap.cc:492] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa: Bootstrap starting.
I20260927 11:18:21.024801 4953 tablet_bootstrap.cc:654] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:21.024983 4955 tablet_bootstrap.cc:492] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351: Bootstrap starting.
I20260927 11:18:21.025283 4954 tablet_bootstrap.cc:492] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140: Bootstrap starting.
I20260927 11:18:21.025410 4953 log.cc:824] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:21.025663 4955 tablet_bootstrap.cc:654] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:21.025875 4954 tablet_bootstrap.cc:654] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:21.026260 4955 log.cc:824] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:21.026396 4954 log.cc:824] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:21.026517 4953 tablet_bootstrap.cc:492] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa: No bootstrap required, opened a new log
I20260927 11:18:21.026587 4953 ts_tablet_manager.cc:1397] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 11:18:21.027009 4955 tablet_bootstrap.cc:492] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351: No bootstrap required, opened a new log
I20260927 11:18:21.027089 4955 ts_tablet_manager.cc:1397] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:18:21.027177 4954 tablet_bootstrap.cc:492] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140: No bootstrap required, opened a new log
I20260927 11:18:21.027244 4954 ts_tablet_manager.cc:1397] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:21.028491 4955 raft_consensus.cc:348] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4d544e3cf54b03b4520af7707fd351" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34361 } } peers { permanent_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39215 } } peers { permanent_uuid: "ac12926cd10f4a9b92b419bc4dfbd140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45139 } }
I20260927 11:18:21.028659 4955 raft_consensus.cc:374] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:21.028703 4955 raft_consensus.cc:729] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4d544e3cf54b03b4520af7707fd351, State: Initialized, Role: FOLLOWER
I20260927 11:18:21.028671 4953 raft_consensus.cc:348] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4d544e3cf54b03b4520af7707fd351" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34361 } } peers { permanent_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39215 } } peers { permanent_uuid: "ac12926cd10f4a9b92b419bc4dfbd140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45139 } }
I20260927 11:18:21.028836 4953 raft_consensus.cc:374] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:21.028831 4955 consensus_queue.cc:260] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4d544e3cf54b03b4520af7707fd351" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34361 } } peers { permanent_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39215 } } peers { permanent_uuid: "ac12926cd10f4a9b92b419bc4dfbd140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45139 } }
I20260927 11:18:21.028877 4953 raft_consensus.cc:729] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6233c7f3a16f43b9a818eee7e224a3aa, State: Initialized, Role: FOLLOWER
I20260927 11:18:21.029035 4953 consensus_queue.cc:260] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4d544e3cf54b03b4520af7707fd351" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34361 } } peers { permanent_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39215 } } peers { permanent_uuid: "ac12926cd10f4a9b92b419bc4dfbd140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45139 } }
I20260927 11:18:21.029145 4955 ts_tablet_manager.cc:1428] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:18:21.029145 4670 heartbeater.cc:503] Master 127.0.98.62:46193 was elected leader, sending a full tablet report...
I20260927 11:18:21.029336 4953 ts_tablet_manager.cc:1428] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 11:18:21.029512 4936 heartbeater.cc:503] Master 127.0.98.62:46193 was elected leader, sending a full tablet report...
I20260927 11:18:21.029735 4954 raft_consensus.cc:348] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4d544e3cf54b03b4520af7707fd351" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34361 } } peers { permanent_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39215 } } peers { permanent_uuid: "ac12926cd10f4a9b92b419bc4dfbd140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45139 } }
I20260927 11:18:21.029886 4954 raft_consensus.cc:374] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:21.029925 4954 raft_consensus.cc:729] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac12926cd10f4a9b92b419bc4dfbd140, State: Initialized, Role: FOLLOWER
I20260927 11:18:21.030035 4954 consensus_queue.cc:260] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4d544e3cf54b03b4520af7707fd351" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34361 } } peers { permanent_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39215 } } peers { permanent_uuid: "ac12926cd10f4a9b92b419bc4dfbd140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45139 } }
I20260927 11:18:21.030344 4954 ts_tablet_manager.cc:1428] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:18:21.030398 4803 heartbeater.cc:503] Master 127.0.98.62:46193 was elected leader, sending a full tablet report...
W20260927 11:18:21.053414 4804 tablet.cc:2369] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:21.115653 4960 raft_consensus.cc:482] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:21.115851 4960 raft_consensus.cc:504] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4d544e3cf54b03b4520af7707fd351" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34361 } } peers { permanent_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39215 } } peers { permanent_uuid: "ac12926cd10f4a9b92b419bc4dfbd140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45139 } }
I20260927 11:18:21.116253 4960 leader_election.cc:288] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e4d544e3cf54b03b4520af7707fd351 (127.0.98.1:34361), ac12926cd10f4a9b92b419bc4dfbd140 (127.0.98.2:45139)
I20260927 11:18:21.119393 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.116566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45639 (local address 127.0.98.2:45139)
0927 11:18:21.116617 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.116618 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.116754 (+ 136us) server_negotiation.cc:407] Connection header received
0927 11:18:21.116807 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.116808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.116822 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.116867 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:21.117111 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.117582 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.118662 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.119042 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.119126 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:21.119135 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.119275 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.119316 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:18:21.119402 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.116619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57957 (local address 127.0.98.1:34361)
0927 11:18:21.116674 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.116676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.116686 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:21.116719 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.116720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.116739 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.116802 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:21.117098 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.117618 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.118641 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.119037 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.119125 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:21.119135 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.119300 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.119335 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 11:18:21.119424 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.116475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34361 (local address 127.0.98.3:57957)
0927 11:18:21.116544 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.116557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:21.116604 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:21.116914 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:21.116915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:21.116916 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:21.116983 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:21.116985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.117725 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.117727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:21.118529 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:21.118531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.119124 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.119125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:21.119217 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:21.119220 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:21.119288 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 11:18:21.119330 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":39}
I20260927 11:18:21.119426 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.116475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45139 (local address 127.0.98.3:45639)
0927 11:18:21.116672 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.116687 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:21.116756 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:21.116883 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:21.116883 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:21.116885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:21.116983 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:21.116985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.117687 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.117688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:21.118569 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:21.118571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.119126 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.119127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:21.119198 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:21.119202 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:21.119269 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 11:18:21.119312 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":28,"thread_start_us":112,"threads_started":1}
I20260927 11:18:21.119814 4624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a79bccbffb453e8a5a002cb5917b4e" candidate_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4d544e3cf54b03b4520af7707fd351" is_pre_election: true
I20260927 11:18:21.120026 4624 raft_consensus.cc:2454] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6233c7f3a16f43b9a818eee7e224a3aa in term 0.
I20260927 11:18:21.120062 4757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a79bccbffb453e8a5a002cb5917b4e" candidate_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac12926cd10f4a9b92b419bc4dfbd140" is_pre_election: true
I20260927 11:18:21.120262 4757 raft_consensus.cc:2454] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6233c7f3a16f43b9a818eee7e224a3aa in term 0.
I20260927 11:18:21.120304 4824 leader_election.cc:302] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6233c7f3a16f43b9a818eee7e224a3aa, 8e4d544e3cf54b03b4520af7707fd351; no voters:
I20260927 11:18:21.120477 4960 raft_consensus.cc:2781] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:21.120546 4960 raft_consensus.cc:482] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:21.120575 4960 raft_consensus.cc:3037] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [term 0 FOLLOWER]: Advancing to term 1
W20260927 11:18:21.142333 4671 tablet.cc:2369] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:21.143725 4960 raft_consensus.cc:504] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4d544e3cf54b03b4520af7707fd351" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34361 } } peers { permanent_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39215 } } peers { permanent_uuid: "ac12926cd10f4a9b92b419bc4dfbd140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45139 } }
I20260927 11:18:21.144047 4960 leader_election.cc:288] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [CANDIDATE]: Term 1 election: Requested vote from peers 8e4d544e3cf54b03b4520af7707fd351 (127.0.98.1:34361), ac12926cd10f4a9b92b419bc4dfbd140 (127.0.98.2:45139)
I20260927 11:18:21.144200 4624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a79bccbffb453e8a5a002cb5917b4e" candidate_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4d544e3cf54b03b4520af7707fd351"
I20260927 11:18:21.144191 4757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a79bccbffb453e8a5a002cb5917b4e" candidate_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac12926cd10f4a9b92b419bc4dfbd140"
I20260927 11:18:21.144301 4624 raft_consensus.cc:3037] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:21.144307 4757 raft_consensus.cc:3037] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:21.145041 4624 raft_consensus.cc:2454] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6233c7f3a16f43b9a818eee7e224a3aa in term 1.
I20260927 11:18:21.145304 4824 leader_election.cc:302] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6233c7f3a16f43b9a818eee7e224a3aa, 8e4d544e3cf54b03b4520af7707fd351; no voters:
I20260927 11:18:21.145444 4960 raft_consensus.cc:2781] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:21.145495 4757 raft_consensus.cc:2454] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6233c7f3a16f43b9a818eee7e224a3aa in term 1.
I20260927 11:18:21.145678 4960 raft_consensus.cc:686] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [term 1 LEADER]: Becoming Leader. State: Replica: 6233c7f3a16f43b9a818eee7e224a3aa, State: Running, Role: LEADER
I20260927 11:18:21.145800 4960 consensus_queue.cc:237] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4d544e3cf54b03b4520af7707fd351" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34361 } } peers { permanent_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39215 } } peers { permanent_uuid: "ac12926cd10f4a9b92b419bc4dfbd140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45139 } }
I20260927 11:18:21.146719 4480 catalog_manager.cc:5495] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 6233c7f3a16f43b9a818eee7e224a3aa (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4d544e3cf54b03b4520af7707fd351" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6233c7f3a16f43b9a818eee7e224a3aa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac12926cd10f4a9b92b419bc4dfbd140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20260927 11:18:21.165380 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.162767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.0.98.3:39215)
0927 11:18:21.162811 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.162813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.162838 (+ 25us) server_negotiation.cc:407] Connection header received
0927 11:18:21.162937 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.162938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.162966 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.163049 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:21.163195 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.163705 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.164545 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:21.164739 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:21.164790 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:21.164930 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:21.165030 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:21.165214 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.165229 (+ 15us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.165250 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:18:21.167315 4757 raft_consensus.cc:1261] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140 [term 1 FOLLOWER]: Refusing update from remote peer 6233c7f3a16f43b9a818eee7e224a3aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:21.167317 4624 raft_consensus.cc:1261] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351 [term 1 FOLLOWER]: Refusing update from remote peer 6233c7f3a16f43b9a818eee7e224a3aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:21.167652 4960 consensus_queue.cc:1035] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e4d544e3cf54b03b4520af7707fd351" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:21.167790 4963 consensus_queue.cc:1035] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac12926cd10f4a9b92b419bc4dfbd140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:21.172219 4479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:21.172315 4479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:21.173032 4479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84dbcef456b846ec9c64a0aa0fdbabbf: Sending DeleteTablet for 3 replicas of tablet 54a79bccbffb453e8a5a002cb5917b4e
I20260927 11:18:21.173377 4604 tablet_service.cc:1514] Processing DeleteTablet for tablet 54a79bccbffb453e8a5a002cb5917b4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54826
I20260927 11:18:21.173418 4870 tablet_service.cc:1514] Processing DeleteTablet for tablet 54a79bccbffb453e8a5a002cb5917b4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47590
I20260927 11:18:21.173597 4737 tablet_service.cc:1514] Processing DeleteTablet for tablet 54a79bccbffb453e8a5a002cb5917b4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38624
I20260927 11:18:21.173810 4978 tablet_replica.cc:330] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140: stopping tablet replica
I20260927 11:18:21.173882 4977 tablet_replica.cc:330] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351: stopping tablet replica
I20260927 11:18:21.173978 4978 raft_consensus.cc:2229] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:21.173996 4977 raft_consensus.cc:2229] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:21.174104 4977 raft_consensus.cc:2258] T 54a79bccbffb453e8a5a002cb5917b4e P 8e4d544e3cf54b03b4520af7707fd351 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:21.174110 4978 raft_consensus.cc:2258] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:21.174196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 4542
I20260927 11:18:21.174322 4976 tablet_replica.cc:330] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa: stopping tablet replica
I20260927 11:18:21.174405 4976 raft_consensus.cc:2229] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:21.174496 4976 raft_consensus.cc:2258] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:21.174778 4978 ts_tablet_manager.cc:1905] T 54a79bccbffb453e8a5a002cb5917b4e P ac12926cd10f4a9b92b419bc4dfbd140: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:21.175086 4976 ts_tablet_manager.cc:1905] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 11:18:21.180641 4465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34361 (error 108)
W20260927 11:18:21.181288 4951 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:21.181130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34361 (local address 127.0.0.1:54838)
0927 11:18:21.181191 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.181238 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34361: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 11:18:21.181406 4465 catalog_manager.cc:4584] TS 8e4d544e3cf54b03b4520af7707fd351 (127.0.98.1:34361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54a79bccbffb453e8a5a002cb5917b4e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34361: connect: Connection reset by peer (error 104)
I20260927 11:18:21.181521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 4675
I20260927 11:18:21.234222 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.233995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34361 (local address 127.0.0.1:54846)
0927 11:18:21.234124 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.234167 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 11:18:21.234630 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.234545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34361 (local address 127.0.0.1:54852)
0927 11:18:21.234580 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.234604 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 11:18:21.235941 4976 ts_tablet_manager.cc:1918] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:21.236039 4976 log.cc:1196] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/54a79bccbffb453e8a5a002cb5917b4e
I20260927 11:18:21.236320 4976 ts_tablet_manager.cc:1939] T 54a79bccbffb453e8a5a002cb5917b4e P 6233c7f3a16f43b9a818eee7e224a3aa: Deleting consensus metadata
I20260927 11:18:21.238706 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.238598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45139 (local address 127.0.0.1:38630)
0927 11:18:21.238648 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.238680 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 11:18:21.238824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 4808
I20260927 11:18:21.252451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 4449
I20260927 11:18:21.252516 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.252380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39215 (local address 127.0.0.1:47604)
0927 11:18:21.252444 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.252477 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
2026-09-27T11:18:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2171 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T11:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:21Z Disabled control of system clock
I20260927 11:18:21.281625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46231
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 11:18:21.362996 4983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:21.363212 4983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:21.363234 4983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:21.364436 4983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:21.364480 4983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:21.364493 4983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:21.364506 4983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:21.364518 4983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:21.366016 4983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46231
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:21.366273 4983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:21.366575 4983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:21.371466 4991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:21.371517 4988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:21.371521 4989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:21.371719 4983 server_base.cc:1034] running on GCE node
I20260927 11:18:21.371966 4983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:21.372217 4983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:21.373361 4983 hybrid_clock.cc:648] HybridClock initialized: now 1790507901373348 us; error 29 us; skew 500 ppm
I20260927 11:18:21.374609 4983 webserver.cc:460] Webserver started at http://127.0.98.62:35733/ using document root <none> and password file <none>
I20260927 11:18:21.374861 4983 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:21.374917 4983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:21.375067 4983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:21.376250 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "98ebb378773a4491b471218175d10e47"
format_stamp: "Formatted at 2026-09-27 11:18:21 on dist-test-slave-lsvg"
server_key: "e10e61c6dc0d6ba76de050b5d59160b4"
server_key_iv: "b1293a274d10ecb814d31a3ed6af9f34"
server_key_version: "encryptionkey@0"
I20260927 11:18:21.376626 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "98ebb378773a4491b471218175d10e47"
format_stamp: "Formatted at 2026-09-27 11:18:21 on dist-test-slave-lsvg"
server_key: "e10e61c6dc0d6ba76de050b5d59160b4"
server_key_iv: "b1293a274d10ecb814d31a3ed6af9f34"
server_key_version: "encryptionkey@0"
I20260927 11:18:21.378065 4983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:21.378834 4997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:21.379009 4983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:21.379108 4983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "98ebb378773a4491b471218175d10e47"
format_stamp: "Formatted at 2026-09-27 11:18:21 on dist-test-slave-lsvg"
server_key: "e10e61c6dc0d6ba76de050b5d59160b4"
server_key_iv: "b1293a274d10ecb814d31a3ed6af9f34"
server_key_version: "encryptionkey@0"
I20260927 11:18:21.379194 4983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:21.382584 4983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:21.382833 4983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:21.382979 4983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:21.386924 4983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46231
I20260927 11:18:21.386955 5049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46231 every 8 connection(s)
I20260927 11:18:21.387269 4983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:21.387888 5050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:21.390679 5050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47: Bootstrap starting.
I20260927 11:18:21.391249 5050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:21.391552 5050 log.cc:824] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:21.392201 5050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47: No bootstrap required, opened a new log
I20260927 11:18:21.393886 5050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ebb378773a4491b471218175d10e47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260927 11:18:21.394037 5050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:21.394083 5050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ebb378773a4491b471218175d10e47, State: Initialized, Role: FOLLOWER
I20260927 11:18:21.394254 5050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ebb378773a4491b471218175d10e47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260927 11:18:21.394339 5050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:21.394389 5050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:21.394439 5050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:21.394976 5050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ebb378773a4491b471218175d10e47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260927 11:18:21.395087 5050 leader_election.cc:302] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ebb378773a4491b471218175d10e47; no voters:
I20260927 11:18:21.395272 5050 leader_election.cc:288] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:21.395309 5053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:21.395493 5053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [term 1 LEADER]: Becoming Leader. State: Replica: 98ebb378773a4491b471218175d10e47, State: Running, Role: LEADER
I20260927 11:18:21.395535 5050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:21.395582 5053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ebb378773a4491b471218175d10e47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } }
I20260927 11:18:21.395916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 4983
I20260927 11:18:21.396039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:21.396251 392 external_mini_cluster.cc:1428] Setting key cb244becf627418d47ca7a9fffbb4a9e
I20260927 11:18:21.397038 5055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ebb378773a4491b471218175d10e47. Latest consensus state: current_term: 1 leader_uuid: "98ebb378773a4491b471218175d10e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ebb378773a4491b471218175d10e47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } } }
I20260927 11:18:21.397153 5055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:21.397330 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98ebb378773a4491b471218175d10e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ebb378773a4491b471218175d10e47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46231 } } }
I20260927 11:18:21.397387 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47 [sys.catalog]: This master's current role is: LEADER
W20260927 11:18:21.397542 5070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:18:21.397608 5070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:18:21.398826 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.396886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41322 (local address 127.0.98.62:46231)
0927 11:18:21.397110 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.397121 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.397232 (+ 111us) server_negotiation.cc:407] Connection header received
0927 11:18:21.397473 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.397482 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.397528 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.397673 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:21.398414 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:21.398418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:21.398418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:21.398422 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:21.398465 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:21.398479 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:21.398554 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.398591 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.398646 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260927 11:18:21.401197 5072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:21.401404 5072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:21.403026 5072 catalog_manager.cc:1347] Generated new cluster ID: 3637ae2a777a4feebf9203d885cc8251
I20260927 11:18:21.403081 5072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:21.429503 5072 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:21.429711 5072 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:21.436775 5072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47: Generated new TSK 0
I20260927 11:18:21.437006 5072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:18:21.445768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46231
--builtin_ntp_servers=127.0.98.20:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 11:18:21.531009 5074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:21.531211 5074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:21.531234 5074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:21.532378 5074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:21.532419 5074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:21.532444 5074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:21.533795 5074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:21.534053 5074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:21.534353 5074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:21.539465 5080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:21.539489 5079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:21.539644 5074 server_base.cc:1034] running on GCE node
W20260927 11:18:21.539474 5082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:21.539951 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:21.540205 5074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:21.541343 5074 hybrid_clock.cc:648] HybridClock initialized: now 1790507901541321 us; error 38 us; skew 500 ppm
I20260927 11:18:21.542527 5074 webserver.cc:460] Webserver started at http://127.0.98.1:42125/ using document root <none> and password file <none>
I20260927 11:18:21.542760 5074 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:21.542805 5074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:21.542989 5074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:21.544057 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "9996c58eabef41b3b8df14b2826a7519"
format_stamp: "Formatted at 2026-09-27 11:18:21 on dist-test-slave-lsvg"
server_key: "ad9f7587dc33ea1773a743f5f6ad0128"
server_key_iv: "fb1bce8be02ce8926aa458f144dcf866"
server_key_version: "encryptionkey@0"
I20260927 11:18:21.544413 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9996c58eabef41b3b8df14b2826a7519"
format_stamp: "Formatted at 2026-09-27 11:18:21 on dist-test-slave-lsvg"
server_key: "ad9f7587dc33ea1773a743f5f6ad0128"
server_key_iv: "fb1bce8be02ce8926aa458f144dcf866"
server_key_version: "encryptionkey@0"
I20260927 11:18:21.545838 5074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:21.546571 5088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:21.546737 5074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:21.546839 5074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "9996c58eabef41b3b8df14b2826a7519"
format_stamp: "Formatted at 2026-09-27 11:18:21 on dist-test-slave-lsvg"
server_key: "ad9f7587dc33ea1773a743f5f6ad0128"
server_key_iv: "fb1bce8be02ce8926aa458f144dcf866"
server_key_version: "encryptionkey@0"
I20260927 11:18:21.546926 5074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:21.550290 5074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:21.550561 5074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:21.550716 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:21.550936 5074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:21.551232 5074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:21.551285 5074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:21.551326 5074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:21.551352 5074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:21.557276 5074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36011
I20260927 11:18:21.557317 5201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36011 every 8 connection(s)
I20260927 11:18:21.557637 5074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:18:21.560709 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.558368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50867 (local address 127.0.98.62:46231)
0927 11:18:21.558444 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.558446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.558637 (+ 191us) server_negotiation.cc:407] Connection header received
0927 11:18:21.559163 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.559164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.559172 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.559218 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:21.560464 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:21.560466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:21.560466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:21.560471 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:21.560489 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:21.560493 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:21.560547 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.560622 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.560640 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:18:21.560927 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.558313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46231 (local address 127.0.98.1:50867)
0927 11:18:21.558514 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.558554 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:21.559048 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:21.559301 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:21.559305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:21.559315 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:21.560284 (+ 969us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:21.560284 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:21.560321 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:21.560357 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:21.560543 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:21.560547 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:21.560550 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:21.560627 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 11:18:21.560687 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260927 11:18:21.560976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 5074
I20260927 11:18:21.561111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:21.561312 392 external_mini_cluster.cc:1428] Setting key 87b55fadf619c03d598d69dfdc872b02
I20260927 11:18:21.561580 5202 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46231
I20260927 11:18:21.561687 5202 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:21.561828 5202 heartbeater.cc:511] Master 127.0.98.62:46231 requested a full tablet report, sending...
I20260927 11:18:21.562322 5014 ts_manager.cc:194] Registered new tserver with Master: 9996c58eabef41b3b8df14b2826a7519 (127.0.98.1:36011)
I20260927 11:18:21.562397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46231
--builtin_ntp_servers=127.0.98.20:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 11:18:21.647454 5205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:21.647708 5205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:21.647750 5205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:21.648977 5205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:21.649044 5205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:21.649139 5205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:21.650658 5205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:21.650923 5205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:21.651224 5205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:21.656386 5211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:21.656502 5213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:21.656665 5210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:21.656688 5205 server_base.cc:1034] running on GCE node
I20260927 11:18:21.656914 5205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:21.657195 5205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:21.658376 5205 hybrid_clock.cc:648] HybridClock initialized: now 1790507901658361 us; error 25 us; skew 500 ppm
I20260927 11:18:21.659696 5205 webserver.cc:460] Webserver started at http://127.0.98.2:34411/ using document root <none> and password file <none>
I20260927 11:18:21.659950 5205 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:21.660030 5205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:21.660182 5205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:21.661327 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "045acd62b1404f569bd3994580a5df90"
format_stamp: "Formatted at 2026-09-27 11:18:21 on dist-test-slave-lsvg"
server_key: "0449df4ce3de74bd9ac2d8e336263d5a"
server_key_iv: "90fe28c6e92200df0f23479217b018fb"
server_key_version: "encryptionkey@0"
I20260927 11:18:21.661686 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "045acd62b1404f569bd3994580a5df90"
format_stamp: "Formatted at 2026-09-27 11:18:21 on dist-test-slave-lsvg"
server_key: "0449df4ce3de74bd9ac2d8e336263d5a"
server_key_iv: "90fe28c6e92200df0f23479217b018fb"
server_key_version: "encryptionkey@0"
I20260927 11:18:21.662992 5205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:18:21.663707 5219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:21.663875 5205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:21.663969 5205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "045acd62b1404f569bd3994580a5df90"
format_stamp: "Formatted at 2026-09-27 11:18:21 on dist-test-slave-lsvg"
server_key: "0449df4ce3de74bd9ac2d8e336263d5a"
server_key_iv: "90fe28c6e92200df0f23479217b018fb"
server_key_version: "encryptionkey@0"
I20260927 11:18:21.664045 5205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:21.667443 5205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:21.667703 5205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:21.667845 5205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:21.668040 5205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:21.668354 5205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:21.668404 5205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:21.668445 5205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:21.668471 5205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:21.674364 5205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36233
I20260927 11:18:21.674398 5332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36233 every 8 connection(s)
I20260927 11:18:21.674743 5205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:18:21.676888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 5205
I20260927 11:18:21.677006 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:21.677311 392 external_mini_cluster.cc:1428] Setting key 2e63f566c9f45e97b0e8f2c91c0c1770
I20260927 11:18:21.678292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46231
--builtin_ntp_servers=127.0.98.20:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 11:18:21.678329 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.675582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52643 (local address 127.0.98.62:46231)
0927 11:18:21.675627 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.675629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.675870 (+ 241us) server_negotiation.cc:407] Connection header received
0927 11:18:21.676496 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.676498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.676508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.676586 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:21.678062 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:21.678063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:21.678063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:21.678067 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:21.678082 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:21.678086 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:21.678139 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.678233 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.678246 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:18:21.678542 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.675545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46231 (local address 127.0.98.2:52643)
0927 11:18:21.675738 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.675778 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:21.676239 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:21.676662 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:21.676666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:21.676674 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:21.677894 (+ 1220us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:21.677895 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:21.677934 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:21.677971 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:21.678150 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:21.678155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:21.678159 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:21.678239 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 11:18:21.678304 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260927 11:18:21.679009 5333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46231
I20260927 11:18:21.679111 5333 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:21.679260 5333 heartbeater.cc:511] Master 127.0.98.62:46231 requested a full tablet report, sending...
I20260927 11:18:21.679625 5014 ts_manager.cc:194] Registered new tserver with Master: 045acd62b1404f569bd3994580a5df90 (127.0.98.2:36233)
W20260927 11:18:21.761703 5336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:21.761919 5336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:21.761940 5336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:21.763118 5336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:21.763161 5336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:21.763196 5336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:21.764544 5336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:21.764801 5336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:21.765161 5336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:21.770217 5341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:21.770287 5344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:21.770246 5342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:21.770601 5336 server_base.cc:1034] running on GCE node
I20260927 11:18:21.770850 5336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:21.771123 5336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:21.772318 5336 hybrid_clock.cc:648] HybridClock initialized: now 1790507901772295 us; error 37 us; skew 500 ppm
I20260927 11:18:21.773751 5336 webserver.cc:460] Webserver started at http://127.0.98.3:46641/ using document root <none> and password file <none>
I20260927 11:18:21.774029 5336 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:21.774085 5336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:21.774286 5336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:21.775468 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "88d7cc6c4f324d559040626dd2bb0bd8"
format_stamp: "Formatted at 2026-09-27 11:18:21 on dist-test-slave-lsvg"
server_key: "3da7792018856590965d8de008048885"
server_key_iv: "0939a7e91b082c3bf8ece76b72f1bed0"
server_key_version: "encryptionkey@0"
I20260927 11:18:21.775872 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "88d7cc6c4f324d559040626dd2bb0bd8"
format_stamp: "Formatted at 2026-09-27 11:18:21 on dist-test-slave-lsvg"
server_key: "3da7792018856590965d8de008048885"
server_key_iv: "0939a7e91b082c3bf8ece76b72f1bed0"
server_key_version: "encryptionkey@0"
I20260927 11:18:21.777345 5336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:21.778134 5350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:21.778456 5336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:21.778563 5336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "88d7cc6c4f324d559040626dd2bb0bd8"
format_stamp: "Formatted at 2026-09-27 11:18:21 on dist-test-slave-lsvg"
server_key: "3da7792018856590965d8de008048885"
server_key_iv: "0939a7e91b082c3bf8ece76b72f1bed0"
server_key_version: "encryptionkey@0"
I20260927 11:18:21.778638 5336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:21.782054 5336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:21.782311 5336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:21.782464 5336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:21.782668 5336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:21.782967 5336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:21.783020 5336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:21.783057 5336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:21.783087 5336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:21.789002 5336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44141
I20260927 11:18:21.789068 5463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44141 every 8 connection(s)
I20260927 11:18:21.789420 5336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:21.792260 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.790145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36779 (local address 127.0.98.62:46231)
0927 11:18:21.790188 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.790190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.790466 (+ 276us) server_negotiation.cc:407] Connection header received
0927 11:18:21.790895 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.790896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.790901 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.790958 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:21.792076 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:21.792077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:21.792077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:21.792081 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:21.792094 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:21.792097 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:21.792146 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.792190 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.792201 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:18:21.792443 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.790127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46231 (local address 127.0.98.3:36779)
0927 11:18:21.790309 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.790343 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:21.790812 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:21.791012 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:21.791015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:21.791023 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:21.791926 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:21.791927 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:21.791966 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:21.791998 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:21.792142 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:21.792145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:21.792147 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:18:21.792197 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 11:18:21.792245 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
I20260927 11:18:21.792891 5464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46231
I20260927 11:18:21.792968 5464 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:21.793133 5464 heartbeater.cc:511] Master 127.0.98.62:46231 requested a full tablet report, sending...
I20260927 11:18:21.793466 5014 ts_manager.cc:194] Registered new tserver with Master: 88d7cc6c4f324d559040626dd2bb0bd8 (127.0.98.3:44141)
I20260927 11:18:21.794732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 5336
I20260927 11:18:21.794849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:21.795060 392 external_mini_cluster.cc:1428] Setting key 178d530a32af4fbabc77a7ca222ea2af
I20260927 11:18:21.795900 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:18:21.798465 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.797261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.0.98.62:46231)
0927 11:18:21.797321 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.797324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.797446 (+ 122us) server_negotiation.cc:407] Connection header received
0927 11:18:21.797515 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.797515 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.797521 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.797571 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:21.798241 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:21.798243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:21.798243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:21.798247 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:21.798264 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:21.798269 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:21.798326 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.798378 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.798394 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:18:21.800094 5014 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:21.801733 5014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:21.805370 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.803738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44141 (local address 127.0.0.1:52518)
0927 11:18:21.803872 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.803916 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:21.803966 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:21.804388 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:21.804395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:21.804401 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:21.804863 (+ 462us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:21.804863 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:21.804880 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:21.804903 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:21.805206 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:21.805210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:21.805212 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:18:21.805288 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 11:18:21.805303 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260927 11:18:21.805400 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.804074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52518 (local address 127.0.98.3:44141)
0927 11:18:21.804243 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.804247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.804255 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:21.804275 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.804277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.804284 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.804381 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:21.805018 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:21.805026 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:21.805027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:21.805031 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:21.805109 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:21.805133 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:21.805213 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.805301 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.805317 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260927 11:18:21.805671 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.803792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36011 (local address 127.0.0.1:38728)
0927 11:18:21.804086 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.804093 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:21.804122 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:21.804410 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:21.804411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:21.804414 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:21.805273 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:21.805273 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:21.805280 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:21.805293 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:21.805545 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:21.805546 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:21.805547 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:18:21.805595 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 11:18:21.805608 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":91,"threads_started":1}
I20260927 11:18:21.805733 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.803898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.0.98.1:36011)
0927 11:18:21.804274 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.804278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.804287 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:21.804311 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.804312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.804319 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.804407 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:21.805385 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:21.805402 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:21.805403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:21.805408 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:21.805440 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:21.805464 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:21.805499 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.805612 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.805642 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":80,"threads_started":1}
I20260927 11:18:21.806454 5136 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5050fe8e2649b39a6e16b134422720 (DEFAULT_TABLE table=test-table [id=63c39f617c9948e8873c8daf6dd94c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:21.806529 5398 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5050fe8e2649b39a6e16b134422720 (DEFAULT_TABLE table=test-table [id=63c39f617c9948e8873c8daf6dd94c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:21.806739 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.803965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36233 (local address 127.0.0.1:44588)
0927 11:18:21.805211 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.805222 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:21.805267 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:21.805730 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:21.805731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:21.805733 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:21.806358 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:21.806359 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:21.806365 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:21.806376 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:21.806617 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:21.806618 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:21.806619 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:18:21.806660 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 11:18:21.806675 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1230,"thread_start_us":75,"threads_started":1}
I20260927 11:18:21.806857 5136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5050fe8e2649b39a6e16b134422720. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:21.806900 5398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5050fe8e2649b39a6e16b134422720. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:21.806931 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.804065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.0.98.2:36233)
0927 11:18:21.804872 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.804876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.805331 (+ 455us) server_negotiation.cc:407] Connection header received
0927 11:18:21.805362 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.805365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.805376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.805448 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:21.806467 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:21.806475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:21.806476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:21.806480 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:21.806515 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:21.806540 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:21.806593 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.806830 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.806846 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":86,"threads_started":1}
I20260927 11:18:21.809669 5478 tablet_bootstrap.cc:492] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8: Bootstrap starting.
I20260927 11:18:21.809901 5267 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5050fe8e2649b39a6e16b134422720 (DEFAULT_TABLE table=test-table [id=63c39f617c9948e8873c8daf6dd94c37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:21.810189 5267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5050fe8e2649b39a6e16b134422720. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:21.810278 5478 tablet_bootstrap.cc:654] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:21.810556 5478 log.cc:824] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:21.810863 5479 tablet_bootstrap.cc:492] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519: Bootstrap starting.
I20260927 11:18:21.811305 5478 tablet_bootstrap.cc:492] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8: No bootstrap required, opened a new log
I20260927 11:18:21.811389 5478 ts_tablet_manager.cc:1397] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:21.811429 5479 tablet_bootstrap.cc:654] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:21.811756 5479 log.cc:824] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:21.813640 5482 tablet_bootstrap.cc:492] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90: Bootstrap starting.
I20260927 11:18:21.813968 5479 tablet_bootstrap.cc:492] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519: No bootstrap required, opened a new log
I20260927 11:18:21.814059 5479 ts_tablet_manager.cc:1397] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 11:18:21.814097 5478 raft_consensus.cc:348] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d7cc6c4f324d559040626dd2bb0bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44141 } } peers { permanent_uuid: "045acd62b1404f569bd3994580a5df90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36233 } } peers { permanent_uuid: "9996c58eabef41b3b8df14b2826a7519" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36011 } }
I20260927 11:18:21.814280 5478 raft_consensus.cc:374] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:21.814319 5478 raft_consensus.cc:729] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d7cc6c4f324d559040626dd2bb0bd8, State: Initialized, Role: FOLLOWER
I20260927 11:18:21.814446 5478 consensus_queue.cc:260] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d7cc6c4f324d559040626dd2bb0bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44141 } } peers { permanent_uuid: "045acd62b1404f569bd3994580a5df90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36233 } } peers { permanent_uuid: "9996c58eabef41b3b8df14b2826a7519" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36011 } }
I20260927 11:18:21.814491 5482 tablet_bootstrap.cc:654] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:21.814708 5478 ts_tablet_manager.cc:1428] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260927 11:18:21.814822 5464 heartbeater.cc:503] Master 127.0.98.62:46231 was elected leader, sending a full tablet report...
I20260927 11:18:21.814831 5482 log.cc:824] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:21.816476 5479 raft_consensus.cc:348] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d7cc6c4f324d559040626dd2bb0bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44141 } } peers { permanent_uuid: "045acd62b1404f569bd3994580a5df90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36233 } } peers { permanent_uuid: "9996c58eabef41b3b8df14b2826a7519" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36011 } }
I20260927 11:18:21.816617 5479 raft_consensus.cc:374] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:21.816639 5479 raft_consensus.cc:729] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9996c58eabef41b3b8df14b2826a7519, State: Initialized, Role: FOLLOWER
I20260927 11:18:21.816726 5479 consensus_queue.cc:260] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d7cc6c4f324d559040626dd2bb0bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44141 } } peers { permanent_uuid: "045acd62b1404f569bd3994580a5df90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36233 } } peers { permanent_uuid: "9996c58eabef41b3b8df14b2826a7519" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36011 } }
I20260927 11:18:21.816962 5479 ts_tablet_manager.cc:1428] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:18:21.816999 5202 heartbeater.cc:503] Master 127.0.98.62:46231 was elected leader, sending a full tablet report...
I20260927 11:18:21.817806 5482 tablet_bootstrap.cc:492] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90: No bootstrap required, opened a new log
I20260927 11:18:21.817886 5482 ts_tablet_manager.cc:1397] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 11:18:21.819752 5482 raft_consensus.cc:348] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d7cc6c4f324d559040626dd2bb0bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44141 } } peers { permanent_uuid: "045acd62b1404f569bd3994580a5df90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36233 } } peers { permanent_uuid: "9996c58eabef41b3b8df14b2826a7519" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36011 } }
I20260927 11:18:21.819880 5482 raft_consensus.cc:374] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:21.819900 5482 raft_consensus.cc:729] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045acd62b1404f569bd3994580a5df90, State: Initialized, Role: FOLLOWER
I20260927 11:18:21.820009 5482 consensus_queue.cc:260] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d7cc6c4f324d559040626dd2bb0bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44141 } } peers { permanent_uuid: "045acd62b1404f569bd3994580a5df90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36233 } } peers { permanent_uuid: "9996c58eabef41b3b8df14b2826a7519" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36011 } }
I20260927 11:18:21.820221 5482 ts_tablet_manager.cc:1428] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:18:21.820304 5333 heartbeater.cc:503] Master 127.0.98.62:46231 was elected leader, sending a full tablet report...
I20260927 11:18:21.898519 5486 raft_consensus.cc:482] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:21.898684 5486 raft_consensus.cc:504] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d7cc6c4f324d559040626dd2bb0bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44141 } } peers { permanent_uuid: "045acd62b1404f569bd3994580a5df90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36233 } } peers { permanent_uuid: "9996c58eabef41b3b8df14b2826a7519" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36011 } }
I20260927 11:18:21.899046 5486 leader_election.cc:288] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88d7cc6c4f324d559040626dd2bb0bd8 (127.0.98.3:44141), 9996c58eabef41b3b8df14b2826a7519 (127.0.98.1:36011)
I20260927 11:18:21.900619 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.899383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36011 (local address 127.0.98.2:54081)
0927 11:18:21.899491 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.899504 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:21.899549 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:21.899712 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:21.899714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:21.899717 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:21.900212 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:21.900212 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:21.900220 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:21.900233 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:21.900467 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:21.900470 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:21.900471 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:18:21.900528 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 11:18:21.900542 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 11:18:21.900689 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.899521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54081 (local address 127.0.98.1:36011)
0927 11:18:21.899560 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.899562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.899573 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:18:21.899613 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.899613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.899617 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.899691 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:21.900338 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:21.900339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:21.900340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:21.900346 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:21.900369 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:21.900376 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:21.900449 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.900546 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.900583 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:18:21.900758 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.899452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59439 (local address 127.0.98.3:44141)
0927 11:18:21.899505 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.899507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.899676 (+ 169us) server_negotiation.cc:407] Connection header received
0927 11:18:21.899710 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.899711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.899717 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.899783 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:21.900497 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:21.900499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:21.900499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:21.900504 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:21.900523 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:21.900527 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:21.900584 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.900663 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.900677 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 11:18:21.900763 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.899383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44141 (local address 127.0.98.2:59439)
0927 11:18:21.899584 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.899593 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:21.899648 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:21.899788 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:21.899789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:21.899791 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:21.900393 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:21.900394 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:21.900401 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:21.900414 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:21.900594 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:21.900594 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:21.900595 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:18:21.900661 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 11:18:21.900674 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":43,"thread_start_us":90,"threads_started":1}
I20260927 11:18:21.900990 5156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5050fe8e2649b39a6e16b134422720" candidate_uuid: "045acd62b1404f569bd3994580a5df90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9996c58eabef41b3b8df14b2826a7519" is_pre_election: true
I20260927 11:18:21.901182 5156 raft_consensus.cc:2454] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045acd62b1404f569bd3994580a5df90 in term 0.
I20260927 11:18:21.901221 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5050fe8e2649b39a6e16b134422720" candidate_uuid: "045acd62b1404f569bd3994580a5df90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88d7cc6c4f324d559040626dd2bb0bd8" is_pre_election: true
I20260927 11:18:21.901393 5418 raft_consensus.cc:2454] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 045acd62b1404f569bd3994580a5df90 in term 0.
I20260927 11:18:21.901422 5222 leader_election.cc:302] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045acd62b1404f569bd3994580a5df90, 9996c58eabef41b3b8df14b2826a7519; no voters:
I20260927 11:18:21.901575 5486 raft_consensus.cc:2781] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:21.901625 5486 raft_consensus.cc:482] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:21.901651 5486 raft_consensus.cc:3037] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:21.902329 5486 raft_consensus.cc:504] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d7cc6c4f324d559040626dd2bb0bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44141 } } peers { permanent_uuid: "045acd62b1404f569bd3994580a5df90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36233 } } peers { permanent_uuid: "9996c58eabef41b3b8df14b2826a7519" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36011 } }
I20260927 11:18:21.902479 5486 leader_election.cc:288] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [CANDIDATE]: Term 1 election: Requested vote from peers 88d7cc6c4f324d559040626dd2bb0bd8 (127.0.98.3:44141), 9996c58eabef41b3b8df14b2826a7519 (127.0.98.1:36011)
I20260927 11:18:21.902616 5156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5050fe8e2649b39a6e16b134422720" candidate_uuid: "045acd62b1404f569bd3994580a5df90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9996c58eabef41b3b8df14b2826a7519"
I20260927 11:18:21.902686 5156 raft_consensus.cc:3037] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:21.903347 5156 raft_consensus.cc:2454] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 045acd62b1404f569bd3994580a5df90 in term 1.
I20260927 11:18:21.903465 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5050fe8e2649b39a6e16b134422720" candidate_uuid: "045acd62b1404f569bd3994580a5df90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88d7cc6c4f324d559040626dd2bb0bd8"
I20260927 11:18:21.903533 5222 leader_election.cc:302] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045acd62b1404f569bd3994580a5df90, 9996c58eabef41b3b8df14b2826a7519; no voters:
I20260927 11:18:21.903553 5418 raft_consensus.cc:3037] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:21.903661 5486 raft_consensus.cc:2781] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:21.903821 5486 raft_consensus.cc:686] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [term 1 LEADER]: Becoming Leader. State: Replica: 045acd62b1404f569bd3994580a5df90, State: Running, Role: LEADER
I20260927 11:18:21.903916 5486 consensus_queue.cc:237] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d7cc6c4f324d559040626dd2bb0bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44141 } } peers { permanent_uuid: "045acd62b1404f569bd3994580a5df90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36233 } } peers { permanent_uuid: "9996c58eabef41b3b8df14b2826a7519" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36011 } }
I20260927 11:18:21.904363 5418 raft_consensus.cc:2454] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 045acd62b1404f569bd3994580a5df90 in term 1.
I20260927 11:18:21.904897 5013 catalog_manager.cc:5495] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 045acd62b1404f569bd3994580a5df90 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "045acd62b1404f569bd3994580a5df90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88d7cc6c4f324d559040626dd2bb0bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "045acd62b1404f569bd3994580a5df90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9996c58eabef41b3b8df14b2826a7519" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36011 } health_report { overall_health: UNKNOWN } } }
I20260927 11:18:21.925216 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.924224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.0.98.2:36233)
0927 11:18:21.924294 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:21.924296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:21.924303 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:18:21.924335 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:21.924336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:21.924349 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:21.924410 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:21.924983 (+ 573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:21.924984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:21.924984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:21.924987 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:21.924998 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:21.925001 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:21.925086 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:21.925151 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 11:18:21.925163 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 11:18:21.925410 5334 tablet.cc:2369] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:21.927002 5418 raft_consensus.cc:1261] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8 [term 1 FOLLOWER]: Refusing update from remote peer 045acd62b1404f569bd3994580a5df90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:21.927029 5156 raft_consensus.cc:1261] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519 [term 1 FOLLOWER]: Refusing update from remote peer 045acd62b1404f569bd3994580a5df90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:21.927431 5486 consensus_queue.cc:1035] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9996c58eabef41b3b8df14b2826a7519" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:21.927597 5488 consensus_queue.cc:1035] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88d7cc6c4f324d559040626dd2bb0bd8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:21.929792 5491 mvcc.cc:205] Tried to move back new op lower bound from 7333920366290460672 to 7333920366199144448. Current Snapshot: MvccSnapshot[applied={T|T < 7333920366290460672}]
I20260927 11:18:21.930145 5492 mvcc.cc:205] Tried to move back new op lower bound from 7333920366290460672 to 7333920366199144448. Current Snapshot: MvccSnapshot[applied={T|T < 7333920366290460672}]
I20260927 11:18:21.930576 5493 mvcc.cc:205] Tried to move back new op lower bound from 7333920366290460672 to 7333920366199144448. Current Snapshot: MvccSnapshot[applied={T|T < 7333920366290460672 or (T in {7333920366290460672})}]
I20260927 11:18:21.932286 5014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:21.932386 5014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:21.933128 5014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98ebb378773a4491b471218175d10e47: Sending DeleteTablet for 3 replicas of tablet 0a5050fe8e2649b39a6e16b134422720
I20260927 11:18:21.933450 5398 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5050fe8e2649b39a6e16b134422720 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:21 UTC) from {username='slave'} at 127.0.0.1:52518
I20260927 11:18:21.933481 5136 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5050fe8e2649b39a6e16b134422720 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:21 UTC) from {username='slave'} at 127.0.0.1:38728
I20260927 11:18:21.933679 5501 tablet_replica.cc:330] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519: stopping tablet replica
I20260927 11:18:21.933676 5267 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5050fe8e2649b39a6e16b134422720 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:21 UTC) from {username='slave'} at 127.0.0.1:44588
I20260927 11:18:21.933779 5501 raft_consensus.cc:2229] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:21.933879 5501 raft_consensus.cc:2258] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:21.933924 5503 tablet_replica.cc:330] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90: stopping tablet replica
I20260927 11:18:21.934098 5503 raft_consensus.cc:2229] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:21.934198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 5074
I20260927 11:18:21.934309 5503 raft_consensus.cc:2258] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:21.934439 5501 ts_tablet_manager.cc:1905] T 0a5050fe8e2649b39a6e16b134422720 P 9996c58eabef41b3b8df14b2826a7519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:21.934821 5503 ts_tablet_manager.cc:1905] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:21.936820 5503 ts_tablet_manager.cc:1918] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:21.936905 5503 log.cc:1196] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/0a5050fe8e2649b39a6e16b134422720
I20260927 11:18:21.937227 5503 ts_tablet_manager.cc:1939] T 0a5050fe8e2649b39a6e16b134422720 P 045acd62b1404f569bd3994580a5df90: Deleting consensus metadata
I20260927 11:18:21.937609 4998 catalog_manager.cc:4841] TS 045acd62b1404f569bd3994580a5df90 (127.0.98.2:36233): tablet 0a5050fe8e2649b39a6e16b134422720 (table test-table [id=63c39f617c9948e8873c8daf6dd94c37]) successfully deleted
W20260927 11:18:21.940873 5000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36011 (error 108)
I20260927 11:18:21.941123 5502 tablet_replica.cc:330] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8: stopping tablet replica
I20260927 11:18:21.941236 5502 raft_consensus.cc:2229] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:21.941356 5502 raft_consensus.cc:2258] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:21.941381 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:21.941254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36011 (local address 127.0.0.1:38736)
0927 11:18:21.941298 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:21.941343 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 11:18:21.941498 5000 catalog_manager.cc:4584] TS 9996c58eabef41b3b8df14b2826a7519 (127.0.98.1:36011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a5050fe8e2649b39a6e16b134422720: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36011: connect: Connection refused (error 111)
I20260927 11:18:21.941534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 5205
I20260927 11:18:21.943025 5502 ts_tablet_manager.cc:1905] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:21.944561 5502 ts_tablet_manager.cc:1918] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:21.944619 5502 log.cc:1196] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/0a5050fe8e2649b39a6e16b134422720
I20260927 11:18:21.944856 5502 ts_tablet_manager.cc:1939] T 0a5050fe8e2649b39a6e16b134422720 P 88d7cc6c4f324d559040626dd2bb0bd8: Deleting consensus metadata
I20260927 11:18:21.945298 4998 catalog_manager.cc:4841] TS 88d7cc6c4f324d559040626dd2bb0bd8 (127.0.98.3:44141): tablet 0a5050fe8e2649b39a6e16b134422720 (table test-table [id=63c39f617c9948e8873c8daf6dd94c37]) successfully deleted
I20260927 11:18:21.946277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 5336
I20260927 11:18:21.950618 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 4983
2026-09-27T11:18:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (696 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 11:18:21.970093 392 mini_oidc.cc:152] Starting JWKS server
I20260927 11:18:21.971130 392 webserver.cc:460] Webserver started at https://127.0.0.1:34181/ using document root <none> and password file <none>
I20260927 11:18:21.971251 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 11:18:21.971433 392 webserver.cc:460] Webserver started at http://127.0.0.1:37201/ using document root <none> and password file <none>
2026-09-27T11:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:21Z Disabled control of system clock
I20260927 11:18:21.979061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44959
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44959
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34181/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/testchainca.cert with env {}
W20260927 11:18:22.060202 5510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:22.060427 5510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:22.060451 5510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:22.061619 5510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:22.061656 5510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:22.061669 5510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:22.061688 5510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:22.061700 5510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 11:18:22.061712 5510 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 11:18:22.061723 5510 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34181/jwks.json
I20260927 11:18:22.063309 5510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44959
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34181/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:22.063515 5510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:22.063688 5510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:22.068578 5518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:22.068775 5510 server_base.cc:1034] running on GCE node
W20260927 11:18:22.068578 5515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:22.068876 5516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:22.069211 5510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:22.069509 5510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:22.070670 5510 hybrid_clock.cc:648] HybridClock initialized: now 1790507902070668 us; error 36 us; skew 500 ppm
I20260927 11:18:22.071977 5510 webserver.cc:460] Webserver started at http://127.0.98.62:34591/ using document root <none> and password file <none>
I20260927 11:18:22.072255 5510 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:22.072336 5510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:22.072481 5510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:22.073629 5510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "11ffadb047b94e0ba90838221c468cd5"
format_stamp: "Formatted at 2026-09-27 11:18:22 on dist-test-slave-lsvg"
server_key: "7cd55d5adc8b6cd7e37fbc6b5ffae266"
server_key_iv: "ce2c3d2f1eb055b53ca40bb5d31cc318"
server_key_version: "encryptionkey@0"
I20260927 11:18:22.073994 5510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "11ffadb047b94e0ba90838221c468cd5"
format_stamp: "Formatted at 2026-09-27 11:18:22 on dist-test-slave-lsvg"
server_key: "7cd55d5adc8b6cd7e37fbc6b5ffae266"
server_key_iv: "ce2c3d2f1eb055b53ca40bb5d31cc318"
server_key_version: "encryptionkey@0"
I20260927 11:18:22.075314 5510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:18:22.076119 5524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:22.076352 5510 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:22.076457 5510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "11ffadb047b94e0ba90838221c468cd5"
format_stamp: "Formatted at 2026-09-27 11:18:22 on dist-test-slave-lsvg"
server_key: "7cd55d5adc8b6cd7e37fbc6b5ffae266"
server_key_iv: "ce2c3d2f1eb055b53ca40bb5d31cc318"
server_key_version: "encryptionkey@0"
I20260927 11:18:22.076527 5510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:22.093151 5510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:22.093492 5510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:22.093658 5510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:22.097441 5510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44959
I20260927 11:18:22.097489 5576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44959 every 8 connection(s)
I20260927 11:18:22.097851 5510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:22.098963 5577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:22.101462 5577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5: Bootstrap starting.
I20260927 11:18:22.102141 5577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:22.102490 5577 log.cc:824] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:22.103215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 5510
I20260927 11:18:22.103348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:22.103343 5577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5: No bootstrap required, opened a new log
I20260927 11:18:22.103528 392 external_mini_cluster.cc:1428] Setting key 56ff7770f6a146fdc955964175d0c84c
I20260927 11:18:22.106202 5577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ffadb047b94e0ba90838221c468cd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260927 11:18:22.106401 5577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:22.106438 5577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ffadb047b94e0ba90838221c468cd5, State: Initialized, Role: FOLLOWER
I20260927 11:18:22.106568 5577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ffadb047b94e0ba90838221c468cd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260927 11:18:22.106647 5577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:22.106678 5577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:22.106722 5577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:22.107410 5577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ffadb047b94e0ba90838221c468cd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260927 11:18:22.107520 5577 leader_election.cc:302] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11ffadb047b94e0ba90838221c468cd5; no voters:
I20260927 11:18:22.107704 5577 leader_election.cc:288] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:22.107733 5582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:22.107841 5582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [term 1 LEADER]: Becoming Leader. State: Replica: 11ffadb047b94e0ba90838221c468cd5, State: Running, Role: LEADER
I20260927 11:18:22.107924 5582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ffadb047b94e0ba90838221c468cd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260927 11:18:22.107988 5577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:22.108251 5582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11ffadb047b94e0ba90838221c468cd5. Latest consensus state: current_term: 1 leader_uuid: "11ffadb047b94e0ba90838221c468cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ffadb047b94e0ba90838221c468cd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } } }
I20260927 11:18:22.108316 5582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:22.108515 5586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:22.108615 5583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11ffadb047b94e0ba90838221c468cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ffadb047b94e0ba90838221c468cd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } } }
I20260927 11:18:22.108708 5583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:22.109082 5586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:22.110736 5586 catalog_manager.cc:1347] Generated new cluster ID: 23ea75bc7efb4cee9c000b6fffb9e170
I20260927 11:18:22.110790 5586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:22.112764 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:22.104032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.0.98.62:44959)
0927 11:18:22.104230 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:22.104237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:22.104261 (+ 24us) server_negotiation.cc:407] Connection header received
0927 11:18:22.104810 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:22.104818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:22.104878 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:22.104992 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:22.105235 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.106173 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.107245 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.107758 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.108210 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:22.111942 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:22.111946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:22.111947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:22.112215 (+ 268us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:22.112266 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:22.112281 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:22.112354 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:22.112465 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 11:18:22.112551 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260927 11:18:22.113711 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:18:22.122112 5586 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:22.122684 5586 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:22.126034 5586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11ffadb047b94e0ba90838221c468cd5: Generated new TSK 0
I20260927 11:18:22.126224 5586 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 11:18:22.152186 5607 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 11:18:22.152310 5607 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 11:18:22.152529 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:22.137236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.0.98.62:44959)
0927 11:18:22.137287 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:22.137289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:22.137403 (+ 114us) server_negotiation.cc:407] Connection header received
0927 11:18:22.137497 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:22.137497 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:22.137507 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:22.137567 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0927 11:18:22.137714 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.138219 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.139373 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.139575 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.139651 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:22.139861 (+ 210us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 11:18:22.152282 (+ 12421us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34181/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 11:18:22.152448 (+ 166us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57202: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34181/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 11:18:22.152609 5581 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57202: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34181/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 11:18:22.153036 5606 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:22.137111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44959 (local address 127.0.0.1:57202)
0927 11:18:22.137315 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:22.137336 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:22.137407 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:22.137572 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:22.137573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:22.137575 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 11:18:22.137644 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:22.137646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.138301 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.138302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:22.139265 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:22.139268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.139652 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.139656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:22.139723 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:22.139746 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 11:18:22.152510 (+ 12764us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34181/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 11:18:22.152634 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44959: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34181/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:44959: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34181/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 11:18:22.153641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 5510
2026-09-27T11:18:22Z chronyd exiting
I20260927 11:18:22.189695 392 test_util.cc:175] -----------------------------------------------
I20260927 11:18:22.189781 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507878147783-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 11:18:22.190501 392 mini_oidc.cc:152] Starting JWKS server
I20260927 11:18:22.191150 392 webserver.cc:460] Webserver started at https://127.0.0.1:36971/ using document root <none> and password file <none>
I20260927 11:18:22.191191 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 11:18:22.191341 392 webserver.cc:460] Webserver started at http://127.0.0.1:46715/ using document root <none> and password file <none>
2026-09-27T11:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:22Z Disabled control of system clock
I20260927 11:18:22.198997 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40605
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40605
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36971/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/testchainca.cert with env {}
W20260927 11:18:22.278403 5613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:22.278679 5613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:22.278720 5613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:22.279933 5613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:22.280009 5613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:22.280030 5613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:22.280052 5613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:22.280083 5613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 11:18:22.280107 5613 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 11:18:22.280131 5613 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36971/jwks.json
I20260927 11:18:22.281770 5613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40605
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36971/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:22.282079 5613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:22.282312 5613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:22.287393 5618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:22.287479 5619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:22.287393 5621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:22.287483 5613 server_base.cc:1034] running on GCE node
I20260927 11:18:22.287833 5613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:22.288130 5613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:22.289294 5613 hybrid_clock.cc:648] HybridClock initialized: now 1790507902289285 us; error 32 us; skew 500 ppm
I20260927 11:18:22.290614 5613 webserver.cc:460] Webserver started at http://127.0.98.62:34205/ using document root <none> and password file <none>
I20260927 11:18:22.290871 5613 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:22.290940 5613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:22.291088 5613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:22.292164 5613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "80ae2f1cfc7c4ce88015ae394e1c09b7"
format_stamp: "Formatted at 2026-09-27 11:18:22 on dist-test-slave-lsvg"
server_key: "31bf4d5e0239e4b5539620420db0785e"
server_key_iv: "9bde4c6112541822a34ec03270cbc4ac"
server_key_version: "encryptionkey@0"
I20260927 11:18:22.292536 5613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "80ae2f1cfc7c4ce88015ae394e1c09b7"
format_stamp: "Formatted at 2026-09-27 11:18:22 on dist-test-slave-lsvg"
server_key: "31bf4d5e0239e4b5539620420db0785e"
server_key_iv: "9bde4c6112541822a34ec03270cbc4ac"
server_key_version: "encryptionkey@0"
I20260927 11:18:22.293932 5613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:22.294687 5627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:22.294864 5613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:22.294965 5613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "80ae2f1cfc7c4ce88015ae394e1c09b7"
format_stamp: "Formatted at 2026-09-27 11:18:22 on dist-test-slave-lsvg"
server_key: "31bf4d5e0239e4b5539620420db0785e"
server_key_iv: "9bde4c6112541822a34ec03270cbc4ac"
server_key_version: "encryptionkey@0"
I20260927 11:18:22.295042 5613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:22.306523 5613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:22.306850 5613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:22.307004 5613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:22.311046 5613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40605
I20260927 11:18:22.311081 5679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40605 every 8 connection(s)
I20260927 11:18:22.311450 5613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:22.312420 5680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:22.313369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 5613
I20260927 11:18:22.313474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:22.313660 392 external_mini_cluster.cc:1428] Setting key 1b9567742813ce9f79bc0a68279a5274
I20260927 11:18:22.315752 5680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7: Bootstrap starting.
I20260927 11:18:22.316388 5680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:22.316709 5680 log.cc:824] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:22.317512 5680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7: No bootstrap required, opened a new log
I20260927 11:18:22.319470 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:22.314219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.0.98.62:40605)
0927 11:18:22.314438 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:22.314445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:22.314471 (+ 26us) server_negotiation.cc:407] Connection header received
0927 11:18:22.314901 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:22.314908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:22.314968 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:22.315084 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:22.315375 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.316302 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.317501 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.317977 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.318078 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:22.318796 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:22.318799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:22.318800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:22.318959 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:22.318995 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:22.319005 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:22.319054 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:22.319157 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 11:18:22.319218 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":110,"threads_started":1}
I20260927 11:18:22.319763 5680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ae2f1cfc7c4ce88015ae394e1c09b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40605 } }
I20260927 11:18:22.319948 5680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:22.319990 5680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ae2f1cfc7c4ce88015ae394e1c09b7, State: Initialized, Role: FOLLOWER
I20260927 11:18:22.320133 5680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ae2f1cfc7c4ce88015ae394e1c09b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40605 } }
I20260927 11:18:22.320241 5680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:22.320276 5680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:22.320325 5680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:22.320564 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:18:22.321043 5680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ae2f1cfc7c4ce88015ae394e1c09b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40605 } }
I20260927 11:18:22.321174 5680 leader_election.cc:302] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80ae2f1cfc7c4ce88015ae394e1c09b7; no voters:
I20260927 11:18:22.321336 5680 leader_election.cc:288] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:22.321363 5685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:22.321476 5685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [term 1 LEADER]: Becoming Leader. State: Replica: 80ae2f1cfc7c4ce88015ae394e1c09b7, State: Running, Role: LEADER
I20260927 11:18:22.321557 5685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ae2f1cfc7c4ce88015ae394e1c09b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40605 } }
I20260927 11:18:22.321591 5680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:22.321904 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80ae2f1cfc7c4ce88015ae394e1c09b7. Latest consensus state: current_term: 1 leader_uuid: "80ae2f1cfc7c4ce88015ae394e1c09b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ae2f1cfc7c4ce88015ae394e1c09b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40605 } } }
I20260927 11:18:22.321913 5686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80ae2f1cfc7c4ce88015ae394e1c09b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80ae2f1cfc7c4ce88015ae394e1c09b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40605 } } }
I20260927 11:18:22.322011 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:22.322016 5686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:22.322314 5690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:22.323046 5690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:22.324572 5690 catalog_manager.cc:1347] Generated new cluster ID: cfc5466bfc4c47c1924ddbf718032f38
I20260927 11:18:22.324626 5690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:22.335619 5690 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:22.336165 5690 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:22.341430 5690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80ae2f1cfc7c4ce88015ae394e1c09b7: Generated new TSK 0
I20260927 11:18:22.341620 5690 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 11:18:22.369572 5710 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 11:18:22.369668 5710 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 11:18:22.369939 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:22.347780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.0.98.62:40605)
0927 11:18:22.347846 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:22.347848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:22.347980 (+ 132us) server_negotiation.cc:407] Connection header received
0927 11:18:22.348051 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:22.348052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:22.348063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:22.348123 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0927 11:18:22.348280 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.348762 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.349942 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.350135 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.350193 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:22.350372 (+ 179us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 11:18:22.369709 (+ 19337us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 11:18:22.369883 (+ 174us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47844: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 11:18:22.370014 5683 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47844: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 11:18:22.370081 5709 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:22.347718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40605 (local address 127.0.0.1:47844)
0927 11:18:22.347898 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:22.347910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:22.347982 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:22.348148 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:22.348149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:22.348151 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 11:18:22.348218 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:22.348221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.348836 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.348837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:22.349828 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:22.349832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.350213 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.350217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:22.350278 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:22.350283 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 11:18:22.369854 (+ 19571us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 11:18:22.369986 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40605: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260927 11:18:22.370409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 5613
2026-09-27T11:18:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (217 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 11:18:22.408046 392 mini_oidc.cc:152] Starting JWKS server
I20260927 11:18:22.408762 392 webserver.cc:460] Webserver started at https://127.0.0.1:34133/ using document root <none> and password file <none>
I20260927 11:18:22.408833 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 11:18:22.408999 392 webserver.cc:460] Webserver started at http://127.0.0.1:45911/ using document root <none> and password file <none>
2026-09-27T11:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:22Z Disabled control of system clock
I20260927 11:18:22.416555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42339
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42339
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34133/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 11:18:22.496847 5716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:22.497087 5716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:22.497110 5716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:22.498310 5716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:22.498360 5716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:22.498373 5716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:22.498386 5716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:22.498399 5716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 11:18:22.498411 5716 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 11:18:22.498463 5716 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34133/jwks.json
I20260927 11:18:22.499807 5716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42339
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34133/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:22.500065 5716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:22.500368 5716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:22.505394 5721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:22.505374 5724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:22.505445 5722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:22.505645 5716 server_base.cc:1034] running on GCE node
I20260927 11:18:22.505895 5716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:22.506176 5716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:22.507344 5716 hybrid_clock.cc:648] HybridClock initialized: now 1790507902507326 us; error 37 us; skew 500 ppm
I20260927 11:18:22.508651 5716 webserver.cc:460] Webserver started at http://127.0.98.62:38063/ using document root <none> and password file <none>
I20260927 11:18:22.508893 5716 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:22.508960 5716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:22.509145 5716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:22.510268 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "277b472c28e444c09f13f83b4017a17e"
format_stamp: "Formatted at 2026-09-27 11:18:22 on dist-test-slave-lsvg"
server_key: "ab228fd164902c4e04deb554f4212721"
server_key_iv: "05ec05fba05aab010d9a56ab9a3fb97b"
server_key_version: "encryptionkey@0"
I20260927 11:18:22.510639 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "277b472c28e444c09f13f83b4017a17e"
format_stamp: "Formatted at 2026-09-27 11:18:22 on dist-test-slave-lsvg"
server_key: "ab228fd164902c4e04deb554f4212721"
server_key_iv: "05ec05fba05aab010d9a56ab9a3fb97b"
server_key_version: "encryptionkey@0"
I20260927 11:18:22.511891 5716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:22.512531 5730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:22.512717 5716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:22.512782 5716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "277b472c28e444c09f13f83b4017a17e"
format_stamp: "Formatted at 2026-09-27 11:18:22 on dist-test-slave-lsvg"
server_key: "ab228fd164902c4e04deb554f4212721"
server_key_iv: "05ec05fba05aab010d9a56ab9a3fb97b"
server_key_version: "encryptionkey@0"
I20260927 11:18:22.512835 5716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:22.533160 5716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:22.533461 5716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:22.533568 5716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:22.537341 5716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42339
I20260927 11:18:22.537421 5782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42339 every 8 connection(s)
I20260927 11:18:22.537751 5716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:22.538479 5783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:22.540930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 5716
I20260927 11:18:22.541018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:22.541103 5783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e: Bootstrap starting.
I20260927 11:18:22.541241 392 external_mini_cluster.cc:1428] Setting key 8108a5fb4eba06642ef49f7ede0b0d0b
I20260927 11:18:22.541819 5783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:22.542124 5783 log.cc:824] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:22.542806 5783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e: No bootstrap required, opened a new log
I20260927 11:18:22.544095 5783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277b472c28e444c09f13f83b4017a17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } }
I20260927 11:18:22.544256 5783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:22.544304 5783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277b472c28e444c09f13f83b4017a17e, State: Initialized, Role: FOLLOWER
I20260927 11:18:22.544433 5783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277b472c28e444c09f13f83b4017a17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } }
I20260927 11:18:22.544524 5783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:22.544567 5783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:22.544616 5783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:22.545418 5783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277b472c28e444c09f13f83b4017a17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } }
I20260927 11:18:22.545536 5783 leader_election.cc:302] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 277b472c28e444c09f13f83b4017a17e; no voters:
I20260927 11:18:22.545707 5783 leader_election.cc:288] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:22.545735 5788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:22.545902 5788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [term 1 LEADER]: Becoming Leader. State: Replica: 277b472c28e444c09f13f83b4017a17e, State: Running, Role: LEADER
I20260927 11:18:22.545985 5783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:22.545994 5788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277b472c28e444c09f13f83b4017a17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } }
I20260927 11:18:22.546356 5788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 277b472c28e444c09f13f83b4017a17e. Latest consensus state: current_term: 1 leader_uuid: "277b472c28e444c09f13f83b4017a17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277b472c28e444c09f13f83b4017a17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } }
I20260927 11:18:22.546440 5788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:22.546648 5789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "277b472c28e444c09f13f83b4017a17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277b472c28e444c09f13f83b4017a17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42339 } } }
I20260927 11:18:22.546722 5789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:22.547037 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:22.541743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.0.98.62:42339)
0927 11:18:22.541987 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:22.541995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:22.542022 (+ 27us) server_negotiation.cc:407] Connection header received
0927 11:18:22.542459 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:22.542466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:22.542522 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:22.542669 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:22.542870 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.543782 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.544735 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.545195 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.545332 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:22.546095 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:22.546099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:22.546099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:22.546309 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:22.546349 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:22.546363 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:22.546405 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:22.546556 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 11:18:22.546660 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
I20260927 11:18:22.547386 5792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:22.547940 5792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:22.547959 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:18:22.549316 5792 catalog_manager.cc:1347] Generated new cluster ID: 35595e74432a4842ad9c6b0b0ddfa185
I20260927 11:18:22.549384 5792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:22.564478 5792 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:22.565006 5792 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:22.570189 5792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 277b472c28e444c09f13f83b4017a17e: Generated new TSK 0
I20260927 11:18:22.570362 5792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:18:22.581604 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:22.577952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.0.98.62:42339)
0927 11:18:22.578000 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:22.578002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:22.578060 (+ 58us) server_negotiation.cc:407] Connection header received
0927 11:18:22.578121 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:22.578122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:22.578134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:22.578210 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0927 11:18:22.578308 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.579093 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.579842 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.580154 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.580225 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:22.580434 (+ 209us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 11:18:22.581394 (+ 960us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34133/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 11:18:22.581531 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34133/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 11:18:22.581660 5786 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34133/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 11:18:22.581686 5812 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:22.577849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42339 (local address 127.0.0.1:50918)
0927 11:18:22.577993 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:22.578003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:22.578064 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:22.578213 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:22.578214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:22.578215 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 11:18:22.578261 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:22.578262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.579165 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.579166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:22.579755 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:22.579757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.580250 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.580258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:22.580307 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:22.580312 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 11:18:22.581492 (+ 1180us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34133/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 11:18:22.581591 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42339: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34133/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260927 11:18:22.581832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 5716
E20260927 11:18:22.592252 5813 webserver.cc:533] Webserver: sslize failed:
E20260927 11:18:22.592322 5813 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T11:18:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T11:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:22Z Disabled control of system clock
I20260927 11:18:22.626467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46589
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46589
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/testchainca.cert with env {}
W20260927 11:18:22.705196 5817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:22.705504 5817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:22.705543 5817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:22.706722 5817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:22.706800 5817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:22.706822 5817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:22.706843 5817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:22.706871 5817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 11:18:22.706893 5817 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 11:18:22.706923 5817 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 11:18:22.708421 5817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:22.708703 5817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:22.708927 5817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:22.713701 5823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:22.713784 5825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:22.713896 5817 server_base.cc:1034] running on GCE node
W20260927 11:18:22.713943 5822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:22.714192 5817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:22.714442 5817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:22.715582 5817 hybrid_clock.cc:648] HybridClock initialized: now 1790507902715566 us; error 35 us; skew 500 ppm
I20260927 11:18:22.716828 5817 webserver.cc:460] Webserver started at http://127.0.98.62:43141/ using document root <none> and password file <none>
I20260927 11:18:22.717028 5817 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:22.717140 5817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:22.717289 5817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:22.718427 5817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "a4c2a720fbac4950bc8fa0146bc47974"
format_stamp: "Formatted at 2026-09-27 11:18:22 on dist-test-slave-lsvg"
server_key: "87b7effa5f856693fdd94ad4adf448f7"
server_key_iv: "bdedd9fe996700e15488261c987ca755"
server_key_version: "encryptionkey@0"
I20260927 11:18:22.718739 5817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4c2a720fbac4950bc8fa0146bc47974"
format_stamp: "Formatted at 2026-09-27 11:18:22 on dist-test-slave-lsvg"
server_key: "87b7effa5f856693fdd94ad4adf448f7"
server_key_iv: "bdedd9fe996700e15488261c987ca755"
server_key_version: "encryptionkey@0"
I20260927 11:18:22.720009 5817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:22.720752 5831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:22.720906 5817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:22.720989 5817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "a4c2a720fbac4950bc8fa0146bc47974"
format_stamp: "Formatted at 2026-09-27 11:18:22 on dist-test-slave-lsvg"
server_key: "87b7effa5f856693fdd94ad4adf448f7"
server_key_iv: "bdedd9fe996700e15488261c987ca755"
server_key_version: "encryptionkey@0"
I20260927 11:18:22.721086 5817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:22.730893 5817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:22.731185 5817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:22.731333 5817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:22.735018 5817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46589
I20260927 11:18:22.735076 5883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46589 every 8 connection(s)
I20260927 11:18:22.735401 5817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:22.736367 5884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:22.738745 5884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974: Bootstrap starting.
I20260927 11:18:22.739326 5884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:22.739605 5884 log.cc:824] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:22.740295 5884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974: No bootstrap required, opened a new log
I20260927 11:18:22.740692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 5817
I20260927 11:18:22.740783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:22.741003 392 external_mini_cluster.cc:1428] Setting key ad9dc5d075af4cb9d7f360fe87de62dd
I20260927 11:18:22.742566 5884 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c2a720fbac4950bc8fa0146bc47974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46589 } }
I20260927 11:18:22.742761 5884 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:22.742800 5884 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4c2a720fbac4950bc8fa0146bc47974, State: Initialized, Role: FOLLOWER
I20260927 11:18:22.742923 5884 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c2a720fbac4950bc8fa0146bc47974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46589 } }
I20260927 11:18:22.743011 5884 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:22.743042 5884 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:22.743083 5884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:22.743808 5884 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c2a720fbac4950bc8fa0146bc47974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46589 } }
I20260927 11:18:22.743914 5884 leader_election.cc:302] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4c2a720fbac4950bc8fa0146bc47974; no voters:
I20260927 11:18:22.744067 5884 leader_election.cc:288] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:22.744132 5889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:22.744297 5884 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:22.744306 5889 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [term 1 LEADER]: Becoming Leader. State: Replica: a4c2a720fbac4950bc8fa0146bc47974, State: Running, Role: LEADER
I20260927 11:18:22.744410 5889 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c2a720fbac4950bc8fa0146bc47974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46589 } }
I20260927 11:18:22.744697 5889 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c2a720fbac4950bc8fa0146bc47974. Latest consensus state: current_term: 1 leader_uuid: "a4c2a720fbac4950bc8fa0146bc47974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c2a720fbac4950bc8fa0146bc47974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46589 } } }
I20260927 11:18:22.744798 5889 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:22.744706 5890 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4c2a720fbac4950bc8fa0146bc47974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4c2a720fbac4950bc8fa0146bc47974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46589 } } }
I20260927 11:18:22.744952 5890 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:22.745137 5896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:22.745806 5896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:22.747416 5896 catalog_manager.cc:1347] Generated new cluster ID: be973661acbd460f830da47fa9c72210
I20260927 11:18:22.747469 5896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:22.748133 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:22.741544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.0.98.62:46589)
0927 11:18:22.741775 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:22.741784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:22.741816 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:18:22.742529 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:22.742538 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:22.742617 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:22.742773 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:22.742994 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.744815 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.745918 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.746512 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.746634 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:22.747425 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:22.747429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:22.747429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:22.747628 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:22.747669 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:22.747684 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:22.747755 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:22.747856 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 11:18:22.747932 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
I20260927 11:18:22.748996 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:18:22.758103 5896 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:22.758594 5896 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:22.764289 5896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4c2a720fbac4950bc8fa0146bc47974: Generated new TSK 0
I20260927 11:18:22.764467 5896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:18:22.926631 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:22.769685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.0.98.62:46589)
0927 11:18:22.769749 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:22.769751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:22.769840 (+ 89us) server_negotiation.cc:407] Connection header received
0927 11:18:22.769910 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:22.769910 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:22.769920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:22.769974 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0927 11:18:22.770108 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.770591 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.771726 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.771898 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.771945 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:22.772173 (+ 228us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 11:18:22.926411 (+154238us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 11:18:22.926571 (+ 160us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33486: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 11:18:22.926726 5888 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33486: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 11:18:22.926972 5913 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:22.769630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46589 (local address 127.0.0.1:33486)
0927 11:18:22.769767 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:22.769778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:22.769844 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:22.769988 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:22.769988 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:22.769991 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0927 11:18:22.770049 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:22.770052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.770675 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.770676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:22.771645 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:22.771648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:22.771986 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:22.771991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:22.772052 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:22.772057 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 11:18:22.926588 (+154531us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 11:18:22.926726 (+ 138us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46589: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":87,"threads_started":1}
I20260927 11:18:22.927335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 5817
2026-09-27T11:18:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (326 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:23 dist-test-slave-lsvg krb5kdc[5925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:23 dist-test-slave-lsvg krb5kdc[5925](info): set up 2 sockets
Sep 27 11:18:23 dist-test-slave-lsvg krb5kdc[5925](info): commencing operation
krb5kdc: starting...
W20260927 11:18:24.153358 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:18:24.272528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41275
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:24.351874 5938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:24.352145 5938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:24.352185 5938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:24.353453 5938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:24.353518 5938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:24.353541 5938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:24.353564 5938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:24.353588 5938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:24.355136 5938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:24.355424 5938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:24.355659 5938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:24.360637 5943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:24.360617 5944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:24.360697 5938 server_base.cc:1034] running on GCE node
W20260927 11:18:24.360613 5946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:24.361037 5938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:24.361413 5938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:24.362569 5938 hybrid_clock.cc:648] HybridClock initialized: now 1790507904362551 us; error 37 us; skew 500 ppm
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:24.365129 5938 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:18:24.365571 5938 webserver.cc:460] Webserver started at http://127.0.98.62:41299/ using document root <none> and password file <none>
I20260927 11:18:24.365757 5938 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:24.365797 5938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:24.365887 5938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:24.366890 5938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "3f09f04bcfb74637ba9dfcf425160aa3"
format_stamp: "Formatted at 2026-09-27 11:18:24 on dist-test-slave-lsvg"
server_key: "97b1b25ce5031dcce6c770fbeda9bc52"
server_key_iv: "2df0f502f01cc9ece9fccdcd4d7240ba"
server_key_version: "encryptionkey@0"
I20260927 11:18:24.367192 5938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f09f04bcfb74637ba9dfcf425160aa3"
format_stamp: "Formatted at 2026-09-27 11:18:24 on dist-test-slave-lsvg"
server_key: "97b1b25ce5031dcce6c770fbeda9bc52"
server_key_iv: "2df0f502f01cc9ece9fccdcd4d7240ba"
server_key_version: "encryptionkey@0"
I20260927 11:18:24.368429 5938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:18:24.369145 5953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:24.369365 5938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:24.369470 5938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "3f09f04bcfb74637ba9dfcf425160aa3"
format_stamp: "Formatted at 2026-09-27 11:18:24 on dist-test-slave-lsvg"
server_key: "97b1b25ce5031dcce6c770fbeda9bc52"
server_key_iv: "2df0f502f01cc9ece9fccdcd4d7240ba"
server_key_version: "encryptionkey@0"
I20260927 11:18:24.369557 5938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:24.381938 5938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:24.384320 5938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:24.384490 5938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:24.388720 5938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41275
I20260927 11:18:24.388751 6005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41275 every 8 connection(s)
I20260927 11:18:24.389281 5938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:24.389912 6006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:24.392163 6006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3: Bootstrap starting.
I20260927 11:18:24.392763 6006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:24.393066 6006 log.cc:824] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:24.393712 6006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3: No bootstrap required, opened a new log
I20260927 11:18:24.395236 6006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f09f04bcfb74637ba9dfcf425160aa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41275 } }
I20260927 11:18:24.395371 6006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:24.395394 6006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f09f04bcfb74637ba9dfcf425160aa3, State: Initialized, Role: FOLLOWER
I20260927 11:18:24.395545 6006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f09f04bcfb74637ba9dfcf425160aa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41275 } }
I20260927 11:18:24.395619 6006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:24.395641 6006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:24.395686 6006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:24.396556 6006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f09f04bcfb74637ba9dfcf425160aa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41275 } }
I20260927 11:18:24.396639 6006 leader_election.cc:302] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f09f04bcfb74637ba9dfcf425160aa3; no voters:
I20260927 11:18:24.396800 6006 leader_election.cc:288] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:24.396850 6009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:24.397097 6009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [term 1 LEADER]: Becoming Leader. State: Replica: 3f09f04bcfb74637ba9dfcf425160aa3, State: Running, Role: LEADER
I20260927 11:18:24.397097 6006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:24.397198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 5938
I20260927 11:18:24.397291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:24.397257 6009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f09f04bcfb74637ba9dfcf425160aa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41275 } }
I20260927 11:18:24.397519 392 external_mini_cluster.cc:1428] Setting key bd9b9876cf2937e6cced5ad1c7839678
I20260927 11:18:24.397642 6011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f09f04bcfb74637ba9dfcf425160aa3. Latest consensus state: current_term: 1 leader_uuid: "3f09f04bcfb74637ba9dfcf425160aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f09f04bcfb74637ba9dfcf425160aa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41275 } } }
I20260927 11:18:24.397724 6011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:24.397652 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f09f04bcfb74637ba9dfcf425160aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f09f04bcfb74637ba9dfcf425160aa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41275 } } }
I20260927 11:18:24.397766 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:24.398492 6018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:24.399047 6018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:24.400601 6018 catalog_manager.cc:1347] Generated new cluster ID: 45bcfdba0cc94aa5b12de13f4fd25752
I20260927 11:18:24.400651 6018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:24.408640 6018 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:24.409255 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:24.398041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.0.98.62:41275)
0927 11:18:24.398385 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:24.398396 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:24.398430 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:18:24.398578 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:24.398678 (+ 100us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:24.398746 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:24.398897 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:24.399616 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.400877 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.401950 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.402555 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.402655 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:24.405424 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:24.405428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:24.405428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:24.405443 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:24.407398 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.407788 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.407791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.407792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.407917 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.408124 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.408125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.408125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.408397 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:24.408559 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:24.408894 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 11:18:24.409026 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":75,"threads_started":1}
I20260927 11:18:24.409355 6018 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:24.411899 6018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f09f04bcfb74637ba9dfcf425160aa3: Generated new TSK 0
I20260927 11:18:24.412056 6018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:18:24.440425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41275
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:24.521806 6032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:24.522027 6032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:24.522049 6032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:24.523200 6032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:24.523240 6032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:24.523267 6032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:24.524878 6032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:24.525135 6032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:24.525370 6032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:24.530153 6040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:24.530126 6037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:24.530154 6038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:24.530385 6032 server_base.cc:1034] running on GCE node
I20260927 11:18:24.530604 6032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:24.530814 6032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:24.531952 6032 hybrid_clock.cc:648] HybridClock initialized: now 1790507904531928 us; error 42 us; skew 500 ppm
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:24.534547 6032 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:18:24.534981 6032 webserver.cc:460] Webserver started at http://127.0.98.1:37027/ using document root <none> and password file <none>
I20260927 11:18:24.535218 6032 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:24.535265 6032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:24.535435 6032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:24.536504 6032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "fedda4c09d53466499f3011d627d5191"
format_stamp: "Formatted at 2026-09-27 11:18:24 on dist-test-slave-lsvg"
server_key: "607266ee2e35dd32bd86a523974dc4c8"
server_key_iv: "01dfa2f8c28c018018a4bd84e2147d29"
server_key_version: "encryptionkey@0"
I20260927 11:18:24.536896 6032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fedda4c09d53466499f3011d627d5191"
format_stamp: "Formatted at 2026-09-27 11:18:24 on dist-test-slave-lsvg"
server_key: "607266ee2e35dd32bd86a523974dc4c8"
server_key_iv: "01dfa2f8c28c018018a4bd84e2147d29"
server_key_version: "encryptionkey@0"
I20260927 11:18:24.538290 6032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:24.539047 6047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:24.539256 6032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:24.539354 6032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "fedda4c09d53466499f3011d627d5191"
format_stamp: "Formatted at 2026-09-27 11:18:24 on dist-test-slave-lsvg"
server_key: "607266ee2e35dd32bd86a523974dc4c8"
server_key_iv: "01dfa2f8c28c018018a4bd84e2147d29"
server_key_version: "encryptionkey@0"
I20260927 11:18:24.539438 6032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:24.545773 6032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:24.547542 6032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:24.547694 6032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:24.547900 6032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:24.548213 6032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:24.548267 6032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:24.548308 6032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:24.548336 6032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:24.553833 6032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39019
I20260927 11:18:24.553907 6160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39019 every 8 connection(s)
I20260927 11:18:24.554214 6032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:18:24.555176 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 6032
I20260927 11:18:24.555286 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:24.555481 392 external_mini_cluster.cc:1428] Setting key 4a584cc4041ff71897ac8f09bd67eee2
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 11:18:24.564725 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:24.554942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51619 (local address 127.0.98.62:41275)
0927 11:18:24.555021 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:24.555024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:24.555810 (+ 786us) server_negotiation.cc:407] Connection header received
0927 11:18:24.556761 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:24.556763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:24.556779 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:24.556835 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:24.557587 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.558076 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.559192 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.559556 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.559639 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:24.561700 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:24.561702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:24.561702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:24.561714 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:24.563364 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.563778 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.563779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.563779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.563838 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.564090 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.564091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.564092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.564209 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:24.564284 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:24.564537 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 11:18:24.564628 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 11:18:24.564929 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:24.554900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41275 (local address 127.0.98.1:51619)
0927 11:18:24.555684 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:24.555720 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:24.556649 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:24.556909 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:24.556913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:24.557234 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:24.557492 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:24.557500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.558165 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.558166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:24.559091 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:24.559094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.559640 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.559641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:24.559737 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:24.560237 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:24.560245 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:24.561601 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:24.563452 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:24.563457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:24.563459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:24.563668 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:24.563907 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:24.563907 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:24.563908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:24.564013 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:24.564275 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:24.564281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:24.564408 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 11:18:24.564515 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 11:18:24.564705 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":85,"threads_started":1}
I20260927 11:18:24.565418 6161 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41275
I20260927 11:18:24.565590 6161 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:24.565776 6161 heartbeater.cc:511] Master 127.0.98.62:41275 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 11:18:24.566306 5970 ts_manager.cc:194] Registered new tserver with Master: fedda4c09d53466499f3011d627d5191 (127.0.98.1:39019)
I20260927 11:18:24.566902 5970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:18:24.582008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41275
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:24.662804 6166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:24.663002 6166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:24.663025 6166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:24.664103 6166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:24.664139 6166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:24.664170 6166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:24.665529 6166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:24.665700 6166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:24.665920 6166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:24.670807 6172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:24.670830 6174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:24.670823 6171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:24.670938 6166 server_base.cc:1034] running on GCE node
I20260927 11:18:24.671288 6166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:24.671487 6166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:24.672619 6166 hybrid_clock.cc:648] HybridClock initialized: now 1790507904672600 us; error 36 us; skew 500 ppm
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:24.675056 6166 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:18:24.675420 6166 webserver.cc:460] Webserver started at http://127.0.98.2:37039/ using document root <none> and password file <none>
I20260927 11:18:24.675598 6166 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:24.675638 6166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:24.675724 6166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:24.676692 6166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "485400b54e8f4218a42a55a8585be9d8"
format_stamp: "Formatted at 2026-09-27 11:18:24 on dist-test-slave-lsvg"
server_key: "990812f14038dcb28a3d7bb0ee06a076"
server_key_iv: "31c299886d44870bd44f80f961692a40"
server_key_version: "encryptionkey@0"
I20260927 11:18:24.676981 6166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "485400b54e8f4218a42a55a8585be9d8"
format_stamp: "Formatted at 2026-09-27 11:18:24 on dist-test-slave-lsvg"
server_key: "990812f14038dcb28a3d7bb0ee06a076"
server_key_iv: "31c299886d44870bd44f80f961692a40"
server_key_version: "encryptionkey@0"
I20260927 11:18:24.678257 6166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:24.678988 6181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:24.679184 6166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:24.679268 6166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "485400b54e8f4218a42a55a8585be9d8"
format_stamp: "Formatted at 2026-09-27 11:18:24 on dist-test-slave-lsvg"
server_key: "990812f14038dcb28a3d7bb0ee06a076"
server_key_iv: "31c299886d44870bd44f80f961692a40"
server_key_version: "encryptionkey@0"
I20260927 11:18:24.679329 6166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:24.686925 6166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:24.688659 6166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:24.688762 6166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:24.688922 6166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:24.689270 6166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:24.689306 6166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:24.689327 6166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:24.689340 6166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:24.694832 6166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36019
I20260927 11:18:24.694916 6294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36019 every 8 connection(s)
I20260927 11:18:24.695209 6166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:18:24.696437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 6166
I20260927 11:18:24.696540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:24.696738 392 external_mini_cluster.cc:1428] Setting key b32238db6a12f698a017519ac42c8a5c
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 11:18:24.706270 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:24.695927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37463 (local address 127.0.98.62:41275)
0927 11:18:24.696003 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:24.696006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:24.696160 (+ 154us) server_negotiation.cc:407] Connection header received
0927 11:18:24.697551 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:24.697552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:24.697565 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:24.697630 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:24.698340 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.699130 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.700462 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.700795 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.700873 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:24.703273 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:24.703275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:24.703275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:24.703284 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:24.704880 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.705315 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.705316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.705316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.705376 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.705609 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.705610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.705611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.705741 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:24.705805 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:24.706092 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 11:18:24.706182 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 11:18:24.706476 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:24.695881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41275 (local address 127.0.98.2:37463)
0927 11:18:24.696034 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:24.696074 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:24.697432 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:24.697701 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:24.697704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:24.697951 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:24.698196 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:24.698204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.699229 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.699230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:24.700374 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:24.700376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.700878 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.700878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:24.700972 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:24.701622 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:24.701630 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:24.703185 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:24.704964 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:24.704969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:24.704970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:24.705240 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:24.705449 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:24.705449 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:24.705450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:24.705540 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:24.705799 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:24.705803 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:24.705943 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 11:18:24.706030 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 11:18:24.706155 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260927 11:18:24.707039 6295 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41275
I20260927 11:18:24.707175 6295 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:24.707377 6295 heartbeater.cc:511] Master 127.0.98.62:41275 requested a full tablet report, sending...
I20260927 11:18:24.707796 5970 ts_manager.cc:194] Registered new tserver with Master: 485400b54e8f4218a42a55a8585be9d8 (127.0.98.2:36019)
I20260927 11:18:24.708221 5970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:18:24.723978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41275
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:24.805764 6300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:24.805968 6300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:24.805991 6300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:24.807169 6300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:24.807214 6300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:24.807269 6300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:24.808825 6300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:24.809128 6300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:24.809360 6300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:24.814410 6306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:24.814419 6308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:24.814419 6305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:24.814739 6300 server_base.cc:1034] running on GCE node
I20260927 11:18:24.814949 6300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:24.815191 6300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:24.816352 6300 hybrid_clock.cc:648] HybridClock initialized: now 1790507904816348 us; error 37 us; skew 500 ppm
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:24.818789 6300 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:18:24.819171 6300 webserver.cc:460] Webserver started at http://127.0.98.3:33791/ using document root <none> and password file <none>
I20260927 11:18:24.819347 6300 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:24.819387 6300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:24.819475 6300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:24.820467 6300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "5dee8bb7915b4352a92a080cd5fa7043"
format_stamp: "Formatted at 2026-09-27 11:18:24 on dist-test-slave-lsvg"
server_key: "0cf759130a2580737a8a3e2606572e4b"
server_key_iv: "d55410767fb91941bd5853eef85f82ae"
server_key_version: "encryptionkey@0"
I20260927 11:18:24.820762 6300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5dee8bb7915b4352a92a080cd5fa7043"
format_stamp: "Formatted at 2026-09-27 11:18:24 on dist-test-slave-lsvg"
server_key: "0cf759130a2580737a8a3e2606572e4b"
server_key_iv: "d55410767fb91941bd5853eef85f82ae"
server_key_version: "encryptionkey@0"
I20260927 11:18:24.822108 6300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:24.822846 6315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:24.823064 6300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:24.823158 6300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "5dee8bb7915b4352a92a080cd5fa7043"
format_stamp: "Formatted at 2026-09-27 11:18:24 on dist-test-slave-lsvg"
server_key: "0cf759130a2580737a8a3e2606572e4b"
server_key_iv: "d55410767fb91941bd5853eef85f82ae"
server_key_version: "encryptionkey@0"
I20260927 11:18:24.823242 6300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:24.835330 6300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:24.837339 6300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:24.837482 6300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:24.837728 6300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:24.838033 6300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:24.838065 6300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:24.838088 6300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:24.838138 6300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:24.844014 6300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44917
I20260927 11:18:24.844110 6428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44917 every 8 connection(s)
I20260927 11:18:24.844420 6300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:24.848631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 6300
I20260927 11:18:24.848726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:24.848932 392 external_mini_cluster.cc:1428] Setting key 26dd7339200faa5950a0140c2c7d0461
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:24.853768 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:24.845192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45701 (local address 127.0.98.62:41275)
0927 11:18:24.845240 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:24.845242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:24.845463 (+ 221us) server_negotiation.cc:407] Connection header received
0927 11:18:24.846392 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:24.846393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:24.846403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:24.846452 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:24.847023 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.847497 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.848457 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.848754 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.848823 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:24.851240 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:24.851241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:24.851242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:24.851248 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:24.852488 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.852876 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.852877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.852877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.852928 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.853214 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.853215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.853215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.853322 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:24.853377 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:24.853626 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 11:18:24.853689 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:18:24.853967 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:24.845172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41275 (local address 127.0.98.3:45701)
0927 11:18:24.845336 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:24.845372 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:24.846299 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:24.846521 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:24.846525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:24.846771 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:24.846957 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:24.846960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.847573 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.847573 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:24.848361 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:24.848363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.848852 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.848853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:24.848959 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:24.849675 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:24.849685 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:24.851164 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:24.852565 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:24.852571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:24.852572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:24.852809 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:24.852986 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:24.852986 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:24.852986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:24.853162 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:24.853385 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:24.853390 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:24.853508 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 11:18:24.853586 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 11:18:24.853724 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260927 11:18:24.854523 6429 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41275
I20260927 11:18:24.854611 6429 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:24.854765 6429 heartbeater.cc:511] Master 127.0.98.62:41275 requested a full tablet report, sending...
I20260927 11:18:24.855101 5970 ts_manager.cc:194] Registered new tserver with Master: 5dee8bb7915b4352a92a080cd5fa7043 (127.0.98.3:44917)
I20260927 11:18:24.855428 5970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45701
I20260927 11:18:24.860705 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/test-admin.keytab.
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:24.877599 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 11:18:24.879271 6436 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:24.878466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.0.98.1:39019)
0927 11:18:24.878634 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:24.878640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:24.878652 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:18:24.878684 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:24.878686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:24.878699 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:24.878783 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:24.879187 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44982: BlockingRecv error: recv got EOF from 127.0.0.1:44982 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
W20260927 11:18:24.879364 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:24.880317 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:24.881290 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:18:24.888814 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:24.881759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.0.98.1:39019)
0927 11:18:24.881815 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:24.881817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:24.881958 (+ 141us) server_negotiation.cc:407] Connection header received
0927 11:18:24.882010 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:24.882010 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:24.882020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:24.882088 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:24.882443 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.883273 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.883971 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.884295 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.884351 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:24.886380 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:24.886386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:24.886387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:24.886393 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:24.887805 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.888103 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.888106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.888106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.888159 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.888321 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.888322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.888322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.888437 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:24.888531 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:24.888595 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 11:18:24.888654 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 11:18:24.890586 6436 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:24.890131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44994 (local address 127.0.98.1:39019)
0927 11:18:24.890167 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:24.890169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:24.890183 (+ 14us) server_negotiation.cc:407] Connection header received
0927 11:18:24.890215 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:24.890215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:24.890223 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:24.890270 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:24.890520 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44994: BlockingRecv error: recv got EOF from 127.0.0.1:44994 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 11:18:24.890739 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/krb5cc)
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:24.891628 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:24.892454 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:18:24.899097 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:24.892904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.0.98.1:39019)
0927 11:18:24.892938 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:24.892940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:24.892970 (+ 30us) server_negotiation.cc:407] Connection header received
0927 11:18:24.893017 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:24.893018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:24.893025 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:24.893148 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:24.893471 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.893990 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.894623 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.894803 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.894851 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:24.896735 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:24.896736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:24.896736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:24.896742 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:24.898135 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.898426 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.898427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.898427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.898467 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.898598 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.898599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.898599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.898671 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:24.898762 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:24.898915 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 11:18:24.898971 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 11:18:24.900355 6436 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:24.899956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45024 (local address 127.0.98.1:39019)
0927 11:18:24.899986 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:24.899988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:24.900052 (+ 64us) server_negotiation.cc:407] Connection header received
0927 11:18:24.900090 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:24.900090 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:24.900097 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:24.900142 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:24.900327 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45024: BlockingRecv error: recv got EOF from 127.0.0.1:45024 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 11:18:24.900520 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507878147783-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:24.901422 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:24.902221 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:18:24.908416 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:24.902730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45034 (local address 127.0.98.1:39019)
0927 11:18:24.902776 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:24.902778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:24.902850 (+ 72us) server_negotiation.cc:407] Connection header received
0927 11:18:24.902901 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:24.902902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:24.902908 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:24.902960 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:24.903323 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.903778 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.904423 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:24.904590 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:24.904635 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:24.906517 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:24.906518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:24.906518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:24.906524 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:24.907628 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.907916 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.907916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.907917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.907960 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:24.908098 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:24.908098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:24.908099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:24.908172 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:24.908233 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:24.908313 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 11:18:24.908362 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 11:18:24.909041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 6032
I20260927 11:18:24.913471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 6166
I20260927 11:18:24.918020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 6300
I20260927 11:18:24.922034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 5938
2026-09-27T11:18:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1817 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[6449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[6449](info): set up 2 sockets
Sep 27 11:18:24 dist-test-slave-lsvg krb5kdc[6449](info): commencing operation
krb5kdc: starting...
W20260927 11:18:25.973335 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:26 dist-test-slave-lsvg krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 11:18:26.092327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42973
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:26.172964 6462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:26.173220 6462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:26.173244 6462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:26.174399 6462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:26.174438 6462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:26.174456 6462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:26.174469 6462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:26.174481 6462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:26.175918 6462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:26.176129 6462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:26.176393 6462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:26.181214 6470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:26.181243 6467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:26.181399 6462 server_base.cc:1034] running on GCE node
W20260927 11:18:26.181474 6468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:26.181749 6462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:26.182050 6462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:26.183218 6462 hybrid_clock.cc:648] HybridClock initialized: now 1790507906183197 us; error 38 us; skew 500 ppm
Sep 27 11:18:26 dist-test-slave-lsvg krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:26.185856 6462 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 11:18:26.186298 6462 webserver.cc:460] Webserver started at http://127.0.98.62:45215/ using document root <none> and password file <none>
I20260927 11:18:26.186533 6462 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:26.186610 6462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:26.186762 6462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:26.187824 6462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "96116c3dd29542429df49b14bbbca4a9"
format_stamp: "Formatted at 2026-09-27 11:18:26 on dist-test-slave-lsvg"
server_key: "88ba820a7cc7015a6ddf897923187f48"
server_key_iv: "223c582f85519d6987f04e1971e6449f"
server_key_version: "encryptionkey@0"
I20260927 11:18:26.188199 6462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "96116c3dd29542429df49b14bbbca4a9"
format_stamp: "Formatted at 2026-09-27 11:18:26 on dist-test-slave-lsvg"
server_key: "88ba820a7cc7015a6ddf897923187f48"
server_key_iv: "223c582f85519d6987f04e1971e6449f"
server_key_version: "encryptionkey@0"
I20260927 11:18:26.189611 6462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:26.190320 6477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:26.190487 6462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:26.190584 6462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "96116c3dd29542429df49b14bbbca4a9"
format_stamp: "Formatted at 2026-09-27 11:18:26 on dist-test-slave-lsvg"
server_key: "88ba820a7cc7015a6ddf897923187f48"
server_key_iv: "223c582f85519d6987f04e1971e6449f"
server_key_version: "encryptionkey@0"
I20260927 11:18:26.190678 6462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:26.222887 6462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:26.225400 6462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:26.225569 6462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:26.229681 6462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42973
I20260927 11:18:26.229732 6529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42973 every 8 connection(s)
I20260927 11:18:26.230125 6462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:26.231110 6530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:26.233559 6530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9: Bootstrap starting.
I20260927 11:18:26.234170 6530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:26.234458 6530 log.cc:824] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:26.235170 6530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9: No bootstrap required, opened a new log
I20260927 11:18:26.236753 6530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } }
I20260927 11:18:26.236898 6530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:26.236922 6530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96116c3dd29542429df49b14bbbca4a9, State: Initialized, Role: FOLLOWER
I20260927 11:18:26.237150 6530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } }
I20260927 11:18:26.237221 6530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:26.237229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 6462
I20260927 11:18:26.237272 6530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:26.237323 6530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:26.237344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:26.237519 392 external_mini_cluster.cc:1428] Setting key a290a82056ed2b7047f5a35309325562
I20260927 11:18:26.238466 6530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } }
I20260927 11:18:26.238572 6530 leader_election.cc:302] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96116c3dd29542429df49b14bbbca4a9; no voters:
I20260927 11:18:26.238731 6530 leader_election.cc:288] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:26.238818 6535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:26.238981 6530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:26.239003 6535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [term 1 LEADER]: Becoming Leader. State: Replica: 96116c3dd29542429df49b14bbbca4a9, State: Running, Role: LEADER
I20260927 11:18:26.239245 6535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } }
I20260927 11:18:26.240518 6536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } }
I20260927 11:18:26.240612 6536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:26.240792 6537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96116c3dd29542429df49b14bbbca4a9. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } }
I20260927 11:18:26.240839 6537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: This master's current role is: LEADER
W20260927 11:18:26.240859 6550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:18:26.240912 6550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:18:26.240957 6552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:26.241129 6552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:26.243499 6552 catalog_manager.cc:1347] Generated new cluster ID: 8089438b1e9f4e119347432a98fdac7d
I20260927 11:18:26.243553 6552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:18:26 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:18:26.249673 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:26.238050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35596 (local address 127.0.98.62:42973)
0927 11:18:26.238282 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:26.238292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:26.238328 (+ 36us) server_negotiation.cc:407] Connection header received
0927 11:18:26.238447 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:26.238455 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:26.238514 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:26.238671 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:26.239503 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:26.241078 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:26.242157 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:26.242767 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:26.242872 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:26.245894 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:26.245899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:26.245899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:26.245913 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:26.247968 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:26.248406 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:26.248409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:26.248409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:26.248523 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:26.248749 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:26.248750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:26.248750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:26.249038 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:26.249206 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:26.249347 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 11:18:26.249478 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":117,"threads_started":1}
I20260927 11:18:26.265025 6552 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:26.265925 6552 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:26.280512 6552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9: Generated new TSK 0
I20260927 11:18:26.280757 6552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 11:18:26.313815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42973
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:26.396157 6556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:26.396382 6556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:26.396409 6556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:26.397574 6556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:26.397624 6556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:26.397650 6556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:26.399045 6556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:26.399226 6556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:26.399412 6556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:18:26.404414 6556 server_base.cc:1034] running on GCE node
W20260927 11:18:26.404582 6562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:26.404670 6564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:26.404613 6561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:26.404909 6556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:26.405216 6556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:26.406355 6556 hybrid_clock.cc:648] HybridClock initialized: now 1790507906406333 us; error 40 us; skew 500 ppm
Sep 27 11:18:26 dist-test-slave-lsvg krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:26.408728 6556 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 11:18:26.409228 6556 webserver.cc:460] Webserver started at http://127.0.98.1:42915/ using document root <none> and password file <none>
I20260927 11:18:26.409451 6556 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:26.409519 6556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:26.409682 6556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:26.410738 6556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "fd95fd6097a444ac9235305e5ab60606"
format_stamp: "Formatted at 2026-09-27 11:18:26 on dist-test-slave-lsvg"
server_key: "7a0017794024a718047c2ef50d629edc"
server_key_iv: "3cb5b1e4465f54ee76dd9576df46fd1f"
server_key_version: "encryptionkey@0"
I20260927 11:18:26.411103 6556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fd95fd6097a444ac9235305e5ab60606"
format_stamp: "Formatted at 2026-09-27 11:18:26 on dist-test-slave-lsvg"
server_key: "7a0017794024a718047c2ef50d629edc"
server_key_iv: "3cb5b1e4465f54ee76dd9576df46fd1f"
server_key_version: "encryptionkey@0"
I20260927 11:18:26.412412 6556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:26.413167 6571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:26.413348 6556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:26.413446 6556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "fd95fd6097a444ac9235305e5ab60606"
format_stamp: "Formatted at 2026-09-27 11:18:26 on dist-test-slave-lsvg"
server_key: "7a0017794024a718047c2ef50d629edc"
server_key_iv: "3cb5b1e4465f54ee76dd9576df46fd1f"
server_key_version: "encryptionkey@0"
I20260927 11:18:26.413533 6556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:26.439795 6556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:26.441979 6556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:26.442153 6556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:26.442401 6556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:26.442783 6556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:26.442839 6556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:26.442881 6556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:26.442924 6556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:26.448954 6556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33883
I20260927 11:18:26.449069 6684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33883 every 8 connection(s)
I20260927 11:18:26.449388 6556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:18:26 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:18:26.458472 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:26.450141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60491 (local address 127.0.98.62:42973)
0927 11:18:26.450215 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:26.450218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:26.450435 (+ 217us) server_negotiation.cc:407] Connection header received
0927 11:18:26.450968 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:26.450969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:26.450986 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:26.451039 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:26.451983 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:26.452495 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:26.453558 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:26.453788 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:26.453876 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:26.455867 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:26.455869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:26.455869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:26.455877 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:26.457143 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:26.457521 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:26.457522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:26.457522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:26.457587 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:26.457834 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:26.457835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:26.457836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:26.457952 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:26.458027 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:26.458296 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 11:18:26.458382 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 11:18:26.458650 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:26.450104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.1:60491)
0927 11:18:26.450304 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:26.450348 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:26.450858 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:26.451119 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:26.451123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:26.451688 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:26.451878 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:26.451882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:26.452600 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:26.452601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:26.453473 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:26.453475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:26.453885 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:26.453887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:26.454009 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:26.454441 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:26.454449 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:26.455771 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:26.457228 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:26.457234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:26.457236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:26.457446 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:26.457654 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:26.457655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:26.457655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:26.457757 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:26.458017 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:26.458022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:26.458158 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 11:18:26.458262 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 11:18:26.458409 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":112,"threads_started":1}
I20260927 11:18:26.459048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 6556
I20260927 11:18:26.459156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:26.459182 6685 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42973
I20260927 11:18:26.459280 6685 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:26.459362 392 external_mini_cluster.cc:1428] Setting key 502a3d536a0e8d322e5604df2748b4f6
I20260927 11:18:26.459481 6685 heartbeater.cc:511] Master 127.0.98.62:42973 requested a full tablet report, sending...
I20260927 11:18:26.460019 6493 ts_manager.cc:194] Registered new tserver with Master: fd95fd6097a444ac9235305e5ab60606 (127.0.98.1:33883)
I20260927 11:18:26.460634 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60491
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 11:18:26.484952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42973
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:26.571399 6690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:26.571609 6690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:26.571632 6690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:26.572707 6690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:26.572741 6690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:26.572767 6690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:26.574245 6690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:26.574426 6690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:26.574597 6690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:26.579742 6696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:26.579751 6698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:26.579751 6695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:26.579778 6690 server_base.cc:1034] running on GCE node
I20260927 11:18:26.580158 6690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:26.580353 6690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:26.581483 6690 hybrid_clock.cc:648] HybridClock initialized: now 1790507906581462 us; error 38 us; skew 500 ppm
Sep 27 11:18:26 dist-test-slave-lsvg krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:26.583995 6690 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 11:18:26.584470 6690 webserver.cc:460] Webserver started at http://127.0.98.2:45151/ using document root <none> and password file <none>
I20260927 11:18:26.584676 6690 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:26.584717 6690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:26.584803 6690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:26.585902 6690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "bf9bc68d3c124f1da0f058f791c3a21f"
format_stamp: "Formatted at 2026-09-27 11:18:26 on dist-test-slave-lsvg"
server_key: "46a20c644c2895ac58382227ec2654f0"
server_key_iv: "9342397d3992f49599bdedf9afce52d1"
server_key_version: "encryptionkey@0"
I20260927 11:18:26.586201 6690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bf9bc68d3c124f1da0f058f791c3a21f"
format_stamp: "Formatted at 2026-09-27 11:18:26 on dist-test-slave-lsvg"
server_key: "46a20c644c2895ac58382227ec2654f0"
server_key_iv: "9342397d3992f49599bdedf9afce52d1"
server_key_version: "encryptionkey@0"
I20260927 11:18:26.587535 6690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:18:26.588196 6705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:26.588363 6690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:26.588444 6690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "bf9bc68d3c124f1da0f058f791c3a21f"
format_stamp: "Formatted at 2026-09-27 11:18:26 on dist-test-slave-lsvg"
server_key: "46a20c644c2895ac58382227ec2654f0"
server_key_iv: "9342397d3992f49599bdedf9afce52d1"
server_key_version: "encryptionkey@0"
I20260927 11:18:26.588496 6690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:26.603224 6690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:26.605566 6690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:26.605674 6690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:26.605856 6690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:26.606237 6690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:26.606288 6690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:26.606323 6690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:26.606354 6690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:26.612979 6690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39715
I20260927 11:18:26.613101 6818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39715 every 8 connection(s)
I20260927 11:18:26.613442 6690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:18:26.614866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 6690
I20260927 11:18:26.614987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:26.615206 392 external_mini_cluster.cc:1428] Setting key 6c88264e6602bf867212080dc60c7eda
Sep 27 11:18:26 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 11:18:26.623927 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:26.614320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37939 (local address 127.0.98.62:42973)
0927 11:18:26.614385 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:26.614387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:26.614706 (+ 319us) server_negotiation.cc:407] Connection header received
0927 11:18:26.615190 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:26.615191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:26.615205 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:26.615283 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:26.616447 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:26.617342 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:26.618385 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:26.618603 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:26.618691 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:26.621138 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:26.621140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:26.621140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:26.621148 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:26.622532 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:26.622938 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:26.622939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:26.622939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:26.623001 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:26.623262 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:26.623263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:26.623264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:26.623386 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:26.623457 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:26.623753 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 11:18:26.623836 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 11:18:26.624126 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:26.614330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.2:37939)
0927 11:18:26.614502 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:26.614537 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:26.615067 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:26.615362 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:26.615367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:26.616103 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:26.616361 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:26.616367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:26.617462 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:26.617463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:26.618277 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:26.618278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:26.618699 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:26.618700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:26.618832 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:26.619393 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:26.619402 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:26.620998 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:26.622632 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:26.622639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:26.622642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:26.622861 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:26.623084 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:26.623085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:26.623085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:26.623195 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:26.623458 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:26.623465 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:26.623616 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 11:18:26.623720 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 11:18:26.623881 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260927 11:18:26.624634 6819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42973
I20260927 11:18:26.624753 6819 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:26.624971 6819 heartbeater.cc:511] Master 127.0.98.62:42973 requested a full tablet report, sending...
I20260927 11:18:26.625423 6493 ts_manager.cc:194] Registered new tserver with Master: bf9bc68d3c124f1da0f058f791c3a21f (127.0.98.2:39715)
I20260927 11:18:26.625936 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260927 11:18:27.461642 6685 heartbeater.cc:503] Master 127.0.98.62:42973 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab.
W20260927 11:18:27.611011 392 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.0.98.3: real 0.995s user 0.002s sys 0.002s
I20260927 11:18:27.611387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42973
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:18:27.626967 6819 heartbeater.cc:503] Master 127.0.98.62:42973 was elected leader, sending a full tablet report...
W20260927 11:18:27.692922 6824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:27.693151 6824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:27.693204 6824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:27.694445 6824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:27.694489 6824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:27.694520 6824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:27.695927 6824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:27.696118 6824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:27.696297 6824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:27.701335 6829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:27.701339 6830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:27.701357 6824 server_base.cc:1034] running on GCE node
W20260927 11:18:27.701339 6832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:27.701769 6824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:27.702049 6824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:27.703217 6824 hybrid_clock.cc:648] HybridClock initialized: now 1790507907703212 us; error 35 us; skew 500 ppm
Sep 27 11:18:27 dist-test-slave-lsvg krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507907, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:27.714013 6824 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 11:18:27.714509 6824 webserver.cc:460] Webserver started at http://127.0.98.3:37553/ using document root <none> and password file <none>
I20260927 11:18:27.714763 6824 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:27.714838 6824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:27.714999 6824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:27.716229 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "4a8c2a0a48624715b798b132579a3ceb"
format_stamp: "Formatted at 2026-09-27 11:18:27 on dist-test-slave-lsvg"
server_key: "fd81d697f9da42f0c086280ca7c9b562"
server_key_iv: "53532e23d55bf46a197b76fe8705a886"
server_key_version: "encryptionkey@0"
I20260927 11:18:27.716602 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4a8c2a0a48624715b798b132579a3ceb"
format_stamp: "Formatted at 2026-09-27 11:18:27 on dist-test-slave-lsvg"
server_key: "fd81d697f9da42f0c086280ca7c9b562"
server_key_iv: "53532e23d55bf46a197b76fe8705a886"
server_key_version: "encryptionkey@0"
I20260927 11:18:27.718152 6824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:18:27.718995 6839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:27.719226 6824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:27.719333 6824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "4a8c2a0a48624715b798b132579a3ceb"
format_stamp: "Formatted at 2026-09-27 11:18:27 on dist-test-slave-lsvg"
server_key: "fd81d697f9da42f0c086280ca7c9b562"
server_key_iv: "53532e23d55bf46a197b76fe8705a886"
server_key_version: "encryptionkey@0"
I20260927 11:18:27.719421 6824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:27.728899 6824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:27.730775 6824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:27.730885 6824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:27.731078 6824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:27.731312 6824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:27.731341 6824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:27.731364 6824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:27.731375 6824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:27.736929 6824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40497
I20260927 11:18:27.737004 6952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40497 every 8 connection(s)
I20260927 11:18:27.737349 6824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 11:18:27 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507907, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:18:27.746307 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.738099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40417 (local address 127.0.98.62:42973)
0927 11:18:27.738230 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:27.738232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:27.738312 (+ 80us) server_negotiation.cc:407] Connection header received
0927 11:18:27.738902 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:27.738903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:27.738914 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:27.739008 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:27.740136 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.740718 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.741772 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.741966 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.742033 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.743964 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:27.743966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:27.743966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:27.743972 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:27.745249 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.745586 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.745586 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.745587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.745636 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.745800 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.745801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.745801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.745896 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:27.745944 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:27.746180 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 11:18:27.746239 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260927 11:18:27.746369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 6824
I20260927 11:18:27.746457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:27.746582 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.738046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.3:40417)
0927 11:18:27.738197 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:27.738237 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:27.738814 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:27.739087 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:27.739091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:27.739810 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:27.740028 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.740033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.740821 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.740822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.741700 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.741702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.742044 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.742045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.742143 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.742578 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:27.742586 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:27.743883 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:27.745313 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.745318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.745320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.745523 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.745675 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.745675 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.745675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.745752 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.745949 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:27.745953 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:27.746071 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 11:18:27.746172 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 11:18:27.746339 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260927 11:18:27.746659 392 external_mini_cluster.cc:1428] Setting key d7abfcbdd3f068daeaac02268de39f48
I20260927 11:18:27.747102 6953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42973
I20260927 11:18:27.747237 6953 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:27.747448 6953 heartbeater.cc:511] Master 127.0.98.62:42973 requested a full tablet report, sending...
I20260927 11:18:27.747810 6494 ts_manager.cc:194] Registered new tserver with Master: 4a8c2a0a48624715b798b132579a3ceb (127.0.98.3:40497)
I20260927 11:18:27.748216 6494 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40417
I20260927 11:18:27.758333 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 11:18:27.783342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33667
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42973,127.0.98.61:33667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:27.863965 6959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:27.864171 6959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:27.864192 6959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:27.865407 6959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:27.865451 6959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:27.865465 6959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:27.865478 6959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:27.865490 6959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:27.866966 6959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42973,127.0.98.61:33667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
W20260927 11:18:27.867101 6959 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:42973,127.0.98.61:33667'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 11:18:27.867254 6959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:27.867491 6959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:27.872324 6965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:27.872473 6959 server_base.cc:1034] running on GCE node
W20260927 11:18:27.872359 6967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:27.872340 6964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:27.872872 6959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:27.873215 6959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:27.874377 6959 hybrid_clock.cc:648] HybridClock initialized: now 1790507907874355 us; error 34 us; skew 500 ppm
Sep 27 11:18:27 dist-test-slave-lsvg krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507907, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:27.876874 6959 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 11:18:27.877384 6959 webserver.cc:460] Webserver started at http://127.0.98.61:35365/ using document root <none> and password file <none>
I20260927 11:18:27.877619 6959 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:27.877692 6959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:27.877835 6959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:27.878875 6959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/data/instance:
uuid: "b6fdfc1f39714f61925ab759c7bfe515"
format_stamp: "Formatted at 2026-09-27 11:18:27 on dist-test-slave-lsvg"
server_key: "479492aa187b95de8dba91c0dad783f8"
server_key_iv: "16457643d5ab28ee7ae5bda24acf6f23"
server_key_version: "encryptionkey@0"
I20260927 11:18:27.879225 6959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/wal/instance:
uuid: "b6fdfc1f39714f61925ab759c7bfe515"
format_stamp: "Formatted at 2026-09-27 11:18:27 on dist-test-slave-lsvg"
server_key: "479492aa187b95de8dba91c0dad783f8"
server_key_iv: "16457643d5ab28ee7ae5bda24acf6f23"
server_key_version: "encryptionkey@0"
I20260927 11:18:27.880508 6959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:27.881234 6974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:27.881403 6959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:27.881515 6959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/wal
uuid: "b6fdfc1f39714f61925ab759c7bfe515"
format_stamp: "Formatted at 2026-09-27 11:18:27 on dist-test-slave-lsvg"
server_key: "479492aa187b95de8dba91c0dad783f8"
server_key_iv: "16457643d5ab28ee7ae5bda24acf6f23"
server_key_version: "encryptionkey@0"
I20260927 11:18:27.881588 6959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:27.897451 6959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:27.899501 6959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:27.899657 6959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:27.903442 6959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33667
I20260927 11:18:27.903491 7026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33667 every 8 connection(s)
I20260927 11:18:27.903841 6959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/data/info.pb
I20260927 11:18:27.904806 7027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:27.906334 7027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:18:27.907918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 6959
Sep 27 11:18:27 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507907, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 11:18:27 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 11:18:27.916446 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.907048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.0.98.62:42973)
0927 11:18:27.907115 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:27.907117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:27.907301 (+ 184us) server_negotiation.cc:407] Connection header received
0927 11:18:27.907734 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:27.907735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:27.907744 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:27.907791 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:27.908629 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.909465 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.910699 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.910985 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.911056 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.913457 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:27.913459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:27.913459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:27.913467 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:27.914998 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.915518 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.915519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.915519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.915570 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.915817 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.915818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.915818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.915950 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:27.916039 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:27.916280 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 11:18:27.916373 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 11:18:27.916687 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.907015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:35608)
0927 11:18:27.907178 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:27.907219 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:27.907655 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:27.907873 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:27.907878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:27.908241 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:27.908524 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.908531 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.909552 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.909553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.910626 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.910627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.911063 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.911064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.911161 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.911658 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:27.911666 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:27.913362 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:27.915074 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.915079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.915080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.915439 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.915632 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.915633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.915633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.915751 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.916015 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:27.916020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:27.916159 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 11:18:27.916237 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 11:18:27.916438 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260927 11:18:27.917337 7027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:18:27.917641 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.908375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55068 (local address 127.0.98.61:33667)
0927 11:18:27.908545 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:27.908549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:27.908566 (+ 17us) server_negotiation.cc:407] Connection header received
0927 11:18:27.908588 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:27.908590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:27.908610 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:27.908680 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:27.909395 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.910604 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.911465 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.911760 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.911847 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.914286 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:27.914292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:27.914292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:27.914298 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:27.915964 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.916381 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.916384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.916384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.916438 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.916787 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.916787 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.916788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.916955 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:27.917130 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:27.917274 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 11:18:27.917535 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
Sep 27 11:18:27 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507907, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 11:18:27.922752 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.917585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33667 (local address 127.0.0.1:55082)
0927 11:18:27.917623 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:27.917633 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:27.917666 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:27.917872 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:27.917873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:27.918001 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:27.918045 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.918046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.918658 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.918659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.919108 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.919109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.919420 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.919421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.919466 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.919789 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:27.919792 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:27.920814 (+ 1022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:27.922078 (+ 1264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.922079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.922079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.922235 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.922369 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.922369 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.922370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.922408 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.922560 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:27.922560 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:27.922589 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 11:18:27.922632 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 11:18:27.922697 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 11:18:27.922873 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.917631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55082 (local address 127.0.98.61:33667)
0927 11:18:27.917710 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:27.917712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:27.917720 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:27.917738 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:27.917738 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:27.917746 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:27.917775 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:27.918126 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.918597 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.919174 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.919362 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.919409 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.920881 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:27.920882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:27.920882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:27.920888 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:27.921978 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.922290 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.922291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.922291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.922333 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.922440 (+ 107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.922440 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.922441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.922525 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:27.922567 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:27.922645 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 11:18:27.922739 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":67,"thread_start_us":74,"threads_started":1}
I20260927 11:18:27.924028 7027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: Bootstrap starting.
I20260927 11:18:27.924816 7027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:27.925153 7027 log.cc:824] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:27.925796 7027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: No bootstrap required, opened a new log
I20260927 11:18:27.927161 7027 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } }
I20260927 11:18:27.927312 7027 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:27.927361 7027 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6fdfc1f39714f61925ab759c7bfe515, State: Initialized, Role: FOLLOWER
I20260927 11:18:27.927496 7027 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } }
I20260927 11:18:27.927754 7034 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } } }
I20260927 11:18:27.927817 7027 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:27.927838 7034 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 11:18:27.929409 7045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:18:27.929491 7045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:18:27.931030 6959 master_runner.cc:428] Detected that this master b6fdfc1f39714f61925ab759c7bfe515 is joining an existing cluster
I20260927 11:18:27.931085 6959 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:33667
I20260927 11:18:27.938791 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.932201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.0.98.61:33667)
0927 11:18:27.932242 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:27.932244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:27.932252 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:27.932305 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:27.932306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:27.932315 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:27.932370 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:27.932898 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.933634 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.934675 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.934848 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.934924 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.935963 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:27.935964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:27.935964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:27.935971 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:27.937654 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.938013 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.938015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.938015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.938073 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.938285 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.938286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.938286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.938399 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:27.938438 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:27.938629 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 11:18:27.938699 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:18:27.939134 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.932037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33667 (local address 127.0.0.1:55092)
0927 11:18:27.932186 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:27.932198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:27.932249 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:27.932404 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:27.932405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:27.932737 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:27.932814 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.932816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.933716 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.933718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.934589 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.934591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.934930 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.934931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.934988 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.935466 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:27.935477 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:27.935875 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:27.937741 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.937742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.937742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.937941 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.938145 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.938146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.938146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.938209 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.938477 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:27.938477 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:27.938521 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 11:18:27.938909 (+ 388us) client_negotiation.cc:240] Negotiation successful
0927 11:18:27.939040 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260927 11:18:27.940006 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.932018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:35624)
0927 11:18:27.932186 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:27.932198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:27.932271 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:27.932421 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:27.932422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:27.932657 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:27.932715 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.932716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.934686 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.934687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.935561 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.935563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.936031 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.936031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.936093 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.936626 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:27.936631 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:27.937123 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:27.938766 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.938767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.938767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.939117 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.939339 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.939340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.939340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.939394 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.939686 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:27.939687 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:27.939741 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 11:18:27.939812 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:18:27.939883 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260927 11:18:27.940035 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.932068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.0.98.62:42973)
0927 11:18:27.932109 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:27.932112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:27.932248 (+ 136us) server_negotiation.cc:407] Connection header received
0927 11:18:27.932351 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:27.932352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:27.932362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:27.932416 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:27.932804 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.933637 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.935652 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.935956 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.936032 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.937227 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:27.937228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:27.937229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:27.937235 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:27.938672 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.939204 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.939206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.939206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.939273 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.939470 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.939471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.939472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.939618 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:27.939687 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:27.939850 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 11:18:27.939932 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 11:18:27 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 11:18:27.948776 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.942254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55106 (local address 127.0.98.61:33667)
0927 11:18:27.942292 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:27.942294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:27.942451 (+ 157us) server_negotiation.cc:407] Connection header received
0927 11:18:27.942544 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:27.942545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:27.942556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:27.942600 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:27.942997 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.943439 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.944201 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.944386 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.944482 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.946405 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:27.946406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:27.946406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:27.946413 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:27.947648 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.948023 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.948024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.948025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.948075 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.948273 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.948274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.948274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.948372 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:27.948442 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:27.948624 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 11:18:27.948694 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:18:27.948804 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.942182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33667 (local address 127.0.0.1:55106)
0927 11:18:27.942321 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:27.942363 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:27.942482 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:27.942613 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:27.942617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:27.942871 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:27.942916 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.942918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.943521 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.943522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.944103 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.944105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.944470 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.944479 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.944571 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.945006 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:27.945010 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:27.946319 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:27.947730 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.947737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.947739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.947931 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.948148 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.948149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.948149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.948212 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.948449 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:27.948455 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:27.948504 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 11:18:27.948600 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 11:18:27.948705 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260927 11:18:27.949355 6494 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:33667: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 33667
}
attrs {
promote: true
}
}
dest_uuid: "96116c3dd29542429df49b14bbbca4a9"
cas_config_opid_index: -1
I20260927 11:18:27.949676 6494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true } }
I20260927 11:18:27.950260 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b6fdfc1f39714f61925ab759c7bfe515 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true } } }
I20260927 11:18:27.950508 7001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:27.951416 7001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 FOLLOWER]: Refusing update from remote peer 96116c3dd29542429df49b14bbbca4a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 11:18:27.951448 7055 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:33667
I20260927 11:18:27.951694 6959 master.cc:537] Master@127.0.98.61:33667 shutting down...
I20260927 11:18:27.951670 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true } } }
I20260927 11:18:27.951743 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:27.951786 7055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
W20260927 11:18:27.953706 6478 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 11:18:27.954167 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 -> Peer b6fdfc1f39714f61925ab759c7bfe515 (127.0.98.61:33667): Couldn't send request to peer b6fdfc1f39714f61925ab759c7bfe515. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 11:18:27.954617 6959 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:27.954695 6959 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:27.954746 6959 tablet_replica.cc:330] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: stopping tablet replica
I20260927 11:18:27.967443 6959 master.cc:552] Master@127.0.98.61:33667 shutdown complete.
I20260927 11:18:27.967502 6959 master_runner.cc:305] Clearing existing system tablet
I20260927 11:18:27.967633 6959 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:27.967682 6959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:27.968392 6959 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260927 11:18:27.968909 7060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:27.969074 6959 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:27.969136 6959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:27.969161 6959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/wal
uuid: "b6fdfc1f39714f61925ab759c7bfe515"
format_stamp: "Formatted at 2026-09-27 11:18:27 on dist-test-slave-lsvg"
server_key: "479492aa187b95de8dba91c0dad783f8"
server_key_iv: "16457643d5ab28ee7ae5bda24acf6f23"
server_key_version: "encryptionkey@0"
I20260927 11:18:27.969929 6959 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:27.971495 6959 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 11:18:27.971565 6959 log.cc:1196] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 11:18:27.971751 6959 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: Deleting consensus metadata
I20260927 11:18:27.971894 6959 master_runner.cc:315] Copying system tablet from 127.0.98.62:42973
I20260927 11:18:27.972537 6959 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42973
I20260927 11:18:27.978906 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.972840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:35630)
0927 11:18:27.972958 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:27.972968 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:27.973082 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:27.973262 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:27.973262 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:27.973423 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:27.973479 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.973481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.974416 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.974416 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.974952 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:27.974953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.975413 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.975414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:27.975459 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.975943 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:27.975947 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:27.976239 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:27.977998 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.977999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.977999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.978157 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.978361 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:27.978361 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:27.978361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:27.978414 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:27.978673 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:27.978674 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:27.978710 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 11:18:27.978760 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 11:18:27.978831 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
I20260927 11:18:27.978978 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:27.972894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.0.98.62:42973)
0927 11:18:27.972944 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:27.972946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:27.973135 (+ 189us) server_negotiation.cc:407] Connection header received
0927 11:18:27.973164 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:27.973165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:27.973176 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:27.973251 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:27.973557 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.974341 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.975038 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:27.975337 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:27.975403 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:27.976323 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:27.976325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:27.976325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:27.976333 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:27.977904 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.978231 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.978232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.978232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.978295 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:27.978474 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:27.978475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:27.978476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:27.978611 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:27.978680 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:27.978783 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 11:18:27.978855 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 11:18:27.979277 6514 tablet_copy_service.cc:140] P 96116c3dd29542429df49b14bbbca4a9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b6fdfc1f39714f61925ab759c7bfe515 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35630)
I20260927 11:18:27.979378 6514 tablet_copy_service.cc:161] P 96116c3dd29542429df49b14bbbca4a9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b6fdfc1f39714f61925ab759c7bfe515 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35630: session id = b6fdfc1f39714f61925ab759c7bfe515-00000000000000000000000000000000
I20260927 11:18:27.980013 6514 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 11:18:27.980656 6959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 11:18:27.981925 6959 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: tablet copy: Starting download of 0 data blocks...
I20260927 11:18:27.982084 6959 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: tablet copy: Starting download of 1 WAL segments...
I20260927 11:18:27.983009 6959 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 11:18:27.983781 6514 tablet_copy_service.cc:342] P 96116c3dd29542429df49b14bbbca4a9: Request end of tablet copy session b6fdfc1f39714f61925ab759c7bfe515-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35630
I20260927 11:18:27.983844 6514 tablet_copy_service.cc:433] P 96116c3dd29542429df49b14bbbca4a9: ending tablet copy session b6fdfc1f39714f61925ab759c7bfe515-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b6fdfc1f39714f61925ab759c7bfe515
I20260927 11:18:27.986384 6959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:27.987668 7067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:27.987720 7070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:27.987679 7068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:27.989197 6959 server_base.cc:1034] running on GCE node
I20260927 11:18:27.989312 6959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:27.989478 6959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:27.990607 6959 hybrid_clock.cc:648] HybridClock initialized: now 1790507907990602 us; error 25 us; skew 500 ppm
Sep 27 11:18:27 dist-test-slave-lsvg krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507907, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:27.991928 6959 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 11:18:27.992254 6959 webserver.cc:460] Webserver started at http://127.0.98.61:45689/ using document root <none> and password file <none>
I20260927 11:18:27.992393 6959 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:27.992434 6959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:27.993121 6959 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:27.993697 7077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:27.993853 6959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:27.993901 6959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/wal
uuid: "b6fdfc1f39714f61925ab759c7bfe515"
format_stamp: "Formatted at 2026-09-27 11:18:27 on dist-test-slave-lsvg"
server_key: "479492aa187b95de8dba91c0dad783f8"
server_key_iv: "16457643d5ab28ee7ae5bda24acf6f23"
server_key_version: "encryptionkey@0"
I20260927 11:18:27.993957 6959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:28.013872 6959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:28.015553 6959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:28.018543 6959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33667
I20260927 11:18:28.018642 7129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33667 every 8 connection(s)
I20260927 11:18:28.018915 6959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-1/data/info.pb
I20260927 11:18:28.019933 7130 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:18:28.020473 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: Bootstrap starting.
I20260927 11:18:28.023342 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:18:28.023584 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: Bootstrap complete.
I20260927 11:18:28.023795 7130 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true } }
I20260927 11:18:28.023885 7130 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6fdfc1f39714f61925ab759c7bfe515, State: Initialized, Role: LEARNER
I20260927 11:18:28.023931 7130 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true } }
I20260927 11:18:28.024111 7130 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:28.024104 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true } } }
I20260927 11:18:28.024304 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: This master's current role is: LEARNER
I20260927 11:18:28.025797 7145 catalog_manager.cc:1259] Loaded cluster ID: 8089438b1e9f4e119347432a98fdac7d
I20260927 11:18:28.025880 7145 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: loading cluster ID for follower catalog manager: success
I20260927 11:18:28.026747 7145 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: acquiring CA information for follower catalog manager: success
I20260927 11:18:28.027029 7145 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 11:18:28.029651 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.019214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.0.98.61:33667)
0927 11:18:28.021859 (+ 2645us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.021861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.021873 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:18:28.021897 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.021898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.021908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.021994 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.022758 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.023813 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.024915 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.025247 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.026884 (+ 1637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.026930 (+ 46us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.026931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.026932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.026936 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.028508 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.028837 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.028838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.028839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.028905 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.029142 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.029143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.029143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.029275 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.029378 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.029458 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.029561 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2621,"thread_start_us":56,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 11:18:28.056768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35285
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42973,127.0.98.61:33667,127.0.98.60:35285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:28.150885 7148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:28.151118 7148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:28.151141 7148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:28.152245 7148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:28.152281 7148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:28.152297 7148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:28.152308 7148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:28.152320 7148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:28.153777 7148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42973,127.0.98.61:33667,127.0.98.60:35285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:28.153967 7148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:28.154136 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:28.158917 7156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:28.158916 7154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:28.158932 7153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:28.158936 7148 server_base.cc:1034] running on GCE node
I20260927 11:18:28.159405 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:28.159685 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:28.160825 7148 hybrid_clock.cc:648] HybridClock initialized: now 1790507908160816 us; error 28 us; skew 500 ppm
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:28.163280 7148 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 11:18:28.163676 7148 webserver.cc:460] Webserver started at http://127.0.98.60:32867/ using document root <none> and password file <none>
I20260927 11:18:28.163848 7148 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:28.163887 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:28.163982 7148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:28.164968 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/data/instance:
uuid: "eb9fefb9d37a451eba23224c55ab3f82"
format_stamp: "Formatted at 2026-09-27 11:18:28 on dist-test-slave-lsvg"
server_key: "e20e65e9fca3c50f8a30dca6691d3c9a"
server_key_iv: "2e73063d39e731b3e0d99959269ba392"
server_key_version: "encryptionkey@0"
I20260927 11:18:28.165345 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/wal/instance:
uuid: "eb9fefb9d37a451eba23224c55ab3f82"
format_stamp: "Formatted at 2026-09-27 11:18:28 on dist-test-slave-lsvg"
server_key: "e20e65e9fca3c50f8a30dca6691d3c9a"
server_key_iv: "2e73063d39e731b3e0d99959269ba392"
server_key_version: "encryptionkey@0"
I20260927 11:18:28.166594 7148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:28.167284 7163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.167470 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:28.167573 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/wal
uuid: "eb9fefb9d37a451eba23224c55ab3f82"
format_stamp: "Formatted at 2026-09-27 11:18:28 on dist-test-slave-lsvg"
server_key: "e20e65e9fca3c50f8a30dca6691d3c9a"
server_key_iv: "2e73063d39e731b3e0d99959269ba392"
server_key_version: "encryptionkey@0"
I20260927 11:18:28.167645 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:28.175491 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:28.177217 7148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:28.177362 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:28.181308 7148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35285
I20260927 11:18:28.181336 7215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35285 every 8 connection(s)
I20260927 11:18:28.181677 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/data/info.pb
I20260927 11:18:28.182072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 7148
I20260927 11:18:28.182765 7216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:28.184916 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:18:28.193997 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.182794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.0.98.60:35285)
0927 11:18:28.182973 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.182983 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.183019 (+ 36us) server_negotiation.cc:407] Connection header received
0927 11:18:28.183558 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.183568 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.183632 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.183789 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.184378 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.185719 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.186744 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.187358 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.187443 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.190059 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.190067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.190067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.190074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.192038 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.192491 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.192494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.192495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.192596 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.192823 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.192824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.192824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.193204 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.193392 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.193579 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.193741 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260927 11:18:28.195339 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.185716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.0.98.62:42973)
0927 11:18:28.185761 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.185763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.185830 (+ 67us) server_negotiation.cc:407] Connection header received
0927 11:18:28.185916 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.185917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.185926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.185970 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.186524 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.187255 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.188389 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.188578 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.188635 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.191393 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.191394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.191395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.191405 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.193007 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.193480 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.193481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.193482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.193556 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.194715 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.194716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.194747 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.194905 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.194973 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.195157 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.195248 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:18:28.195355 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.185581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:35640)
0927 11:18:28.185728 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.185766 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.185834 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.186006 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.186008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.186337 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.186436 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.186437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.187346 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.187347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.188293 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.188296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.188665 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.188666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.188755 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.189344 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.189352 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.191291 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.193145 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.193151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.193153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.193402 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.193622 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.193623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.193623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.193686 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.194965 (+ 1279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.194970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.195024 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.195129 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.195247 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":99,"threads_started":1}
I20260927 11:18:28.195832 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 11:18:28.202468 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.196087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33667 (local address 127.0.0.1:55134)
0927 11:18:28.196138 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.196149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.196188 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.196324 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.196325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.196461 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.196508 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.196510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.197234 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.197235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.197744 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.197746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.198154 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.198154 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.198222 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.198668 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.198673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.199790 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.201480 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.201482 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.201482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.201648 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.201848 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.201849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.201850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.201904 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.202157 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.202158 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.202199 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.202286 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.202380 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 11:18:28.202497 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.196182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55134 (local address 127.0.98.61:33667)
0927 11:18:28.196222 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.196225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.196233 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:28.196255 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.196256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.196265 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.196321 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.196585 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.197137 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.197839 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.198060 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.198154 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.199874 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.199875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.199876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.199883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.201383 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.201719 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.201720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.201721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.201775 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.201980 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.201981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.201982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.202093 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.202165 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.202311 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.202406 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:18:28.202950 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 11:18:28.208592 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.203192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35285 (local address 127.0.0.1:41024)
0927 11:18:28.203236 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.203247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.203281 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.203405 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.203405 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.203512 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.203538 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.203540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.204101 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.204102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.204552 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.204554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.204873 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.204873 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.204907 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.205347 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.205350 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.206406 (+ 1056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.207893 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.207894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.207894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.208045 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.208185 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.208186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.208186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.208225 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.208374 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.208375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.208404 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.208447 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.208515 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 11:18:28.208642 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.203252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.0.98.60:35285)
0927 11:18:28.203286 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.203288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.203297 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:28.203340 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.203341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.203347 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.203401 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.203593 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.204041 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.204622 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.204812 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.204864 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.206478 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.206479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.206479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.206486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.207830 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.208096 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.208096 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.208097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.208137 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.208258 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.208258 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.208259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.208341 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.208382 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.208464 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.208553 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":15}
I20260927 11:18:28.210029 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: Bootstrap starting.
I20260927 11:18:28.210875 7216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:28.211158 7216 log.cc:824] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:28.211906 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: No bootstrap required, opened a new log
I20260927 11:18:28.213390 7216 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } }
I20260927 11:18:28.213551 7216 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:28.213599 7216 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9fefb9d37a451eba23224c55ab3f82, State: Initialized, Role: FOLLOWER
I20260927 11:18:28.213743 7216 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } }
I20260927 11:18:28.214064 7216 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:28.214035 7221 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } } }
I20260927 11:18:28.214134 7221 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 11:18:28.215610 7232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:18:28.215669 7232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:18:28.217922 7148 master_runner.cc:428] Detected that this master eb9fefb9d37a451eba23224c55ab3f82 is joining an existing cluster
I20260927 11:18:28.217981 7148 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:35285
I20260927 11:18:28.226266 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.219346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.0.98.61:33667)
0927 11:18:28.219404 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.219406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.219413 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:18:28.219441 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.219442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.219450 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.219527 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.220216 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.220957 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.222203 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.222491 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.222526 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.223620 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.223622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.223622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.223629 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.225124 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.225461 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.225462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.225463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.225527 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.225760 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.225761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.225761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.225873 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.225945 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.226099 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.226173 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 11:18:28.226284 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.218923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33667 (local address 127.0.0.1:55148)
0927 11:18:28.219080 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.219092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.219141 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.219618 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.219619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.220028 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.220122 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.220124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.221079 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.221080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.222120 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.222122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.222615 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.222615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.222674 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.223173 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.223178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.223511 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.225189 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.225190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.225190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.225374 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.225630 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.225631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.225631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.225695 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.225938 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.225939 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.225989 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.226075 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.226174 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":52,"thread_start_us":103,"threads_started":1}
I20260927 11:18:28.245540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 6556
I20260927 11:18:28.250408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33883
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42973
--webserver_port=42915
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42973,127.0.98.61:33667
--tserver_master_addrs=127.0.98.62:42973,127.0.98.61:33667,127.0.98.60:35285 with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:18:28.257705 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.219337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.0.98.60:35285)
0927 11:18:28.219371 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.219373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.219478 (+ 105us) server_negotiation.cc:407] Connection header received
0927 11:18:28.219523 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.219524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.219532 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.219584 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.220208 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.220915 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.221990 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.222263 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.222696 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.254546 (+ 31850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.254548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.254548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.254556 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.256489 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.256889 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.256890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.256890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.256931 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.257170 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.257171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.257171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.257299 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.257343 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.257518 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.257605 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:18:28.257763 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.218950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.0.98.62:42973)
0927 11:18:28.219003 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.219005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.219370 (+ 365us) server_negotiation.cc:407] Connection header received
0927 11:18:28.219652 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.219653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.219666 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.219698 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.220217 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.220964 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.222170 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.222459 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.222512 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.254569 (+ 32057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.254571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.254572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.254580 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.256378 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.256871 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.256872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.256872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.256929 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.257162 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.257163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.257164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.257299 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.257358 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.257598 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.257673 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:18:28.258278 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.219166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35285 (local address 127.0.0.1:41040)
0927 11:18:28.219411 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.219419 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.219474 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.219589 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.219590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.220022 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.220124 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.220126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.221013 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.221014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.221895 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.221898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.222765 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.222766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.222821 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.253846 (+ 31025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.253850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.254434 (+ 584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.256563 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.256564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.256564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.256818 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.257016 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.257017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.257017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.257102 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.257382 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.257383 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.257419 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.258014 (+ 595us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.258131 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260927 11:18:28.258358 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.219015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:35656)
0927 11:18:28.219310 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.219320 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.219569 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.219769 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.219769 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.220021 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.220123 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.220124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.221079 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.221080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.222071 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.222073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.222613 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.222615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.222693 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.253773 (+ 31080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.253779 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.254434 (+ 655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.256499 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.256501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.256501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.256789 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.256998 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.256999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.257000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.257090 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.257382 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.257383 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.257429 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.257495 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.258190 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":657,"thread_start_us":65,"threads_started":1}
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 11:18:28.267059 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.260709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.0.98.60:35285)
0927 11:18:28.260743 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.260745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.260754 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:28.260833 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.260833 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.260842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.260897 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.261241 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.261697 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.262382 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.262590 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.262677 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.264456 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.264457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.264458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.264465 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.265928 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.266293 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.266294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.266294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.266344 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.266551 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.266553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.266553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.266672 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.266746 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.266899 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.266970 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:18:28.267068 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.260668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35285 (local address 127.0.0.1:41056)
0927 11:18:28.260713 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.260723 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.260762 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.260901 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.260902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.261103 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.261157 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.261159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.261785 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.261786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.262304 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.262305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.262677 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.262678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.262749 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.263166 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.263171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.264363 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.266036 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.266037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.266037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.266207 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.266421 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.266422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.266422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.266481 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.266740 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.266741 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.266786 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.266873 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.266979 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 11:18:28.267871 6494 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:35285: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 35285
}
attrs {
promote: true
}
}
dest_uuid: "96116c3dd29542429df49b14bbbca4a9"
cas_config_opid_index: 5
I20260927 11:18:28.268149 6494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } }
I20260927 11:18:28.268476 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER eb9fefb9d37a451eba23224c55ab3f82 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } } }
I20260927 11:18:28.269811 7190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:28.269997 7055 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:35285
I20260927 11:18:28.270069 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } } }
I20260927 11:18:28.270131 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:28.270416 7148 master.cc:537] Master@127.0.98.60:35285 shutting down...
I20260927 11:18:28.270840 7190 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 1 FOLLOWER]: Refusing update from remote peer 96116c3dd29542429df49b14bbbca4a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 11:18:28.271138 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 11:18:28.271937 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 -> Peer eb9fefb9d37a451eba23224c55ab3f82 (127.0.98.60:35285): Couldn't send request to peer eb9fefb9d37a451eba23224c55ab3f82. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 11:18:28.274145 7148 raft_consensus.cc:2229] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:28.274240 7148 raft_consensus.cc:2258] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:28.274296 7148 tablet_replica.cc:330] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: stopping tablet replica
I20260927 11:18:28.276221 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.269423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.0.98.61:33667)
0927 11:18:28.269462 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.269464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.271062 (+ 1598us) server_negotiation.cc:407] Connection header received
0927 11:18:28.272214 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.272215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.272239 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.272294 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:28.272682 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.273483 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.275072 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.275704 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.275750 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.275837 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.276090 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.276138 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:18:28.276572 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.269322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33667 (local address 127.0.0.1:55158)
0927 11:18:28.271002 (+ 1680us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.271014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.272134 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.272540 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.272541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.272543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:28.272589 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.272591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.273598 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.273599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.274971 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.274975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.275902 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.275903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.275977 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.276004 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.276426 (+ 422us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.276485 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1660}
I20260927 11:18:28.276934 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 LEARNER]: Refusing update from remote peer 96116c3dd29542429df49b14bbbca4a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 11:18:28.277163 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 11:18:28.277715 7148 master.cc:552] Master@127.0.98.60:35285 shutdown complete.
I20260927 11:18:28.277751 7148 master_runner.cc:305] Clearing existing system tablet
I20260927 11:18:28.277719 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER eb9fefb9d37a451eba23224c55ab3f82 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } } }
I20260927 11:18:28.277874 7148 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:28.277915 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:28.278833 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } } }
I20260927 11:18:28.278923 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: This master's current role is: LEARNER
I20260927 11:18:28.279075 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96116c3dd29542429df49b14bbbca4a9. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: true } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } } }
I20260927 11:18:28.279175 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: This master's current role is: LEARNER
I20260927 11:18:28.281533 7148 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260927 11:18:28.281560 7056 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9: attempting to promote NON_VOTER b6fdfc1f39714f61925ab759c7bfe515 to VOTER
I20260927 11:18:28.281778 7056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } }
I20260927 11:18:28.282362 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 LEARNER]: Refusing update from remote peer 96116c3dd29542429df49b14bbbca4a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 11:18:28.282559 7245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 11:18:28.282625 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.282202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35285 (local address 127.0.0.1:41072)
0927 11:18:28.282557 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.282601 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338}
I20260927 11:18:28.282702 7246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.282794 7148 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:28.282845 7148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:18:28.282872 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/wal
uuid: "eb9fefb9d37a451eba23224c55ab3f82"
format_stamp: "Formatted at 2026-09-27 11:18:28 on dist-test-slave-lsvg"
server_key: "e20e65e9fca3c50f8a30dca6691d3c9a"
server_key_iv: "2e73063d39e731b3e0d99959269ba392"
server_key_version: "encryptionkey@0"
I20260927 11:18:28.282991 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.282907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35285 (local address 127.0.0.1:41080)
0927 11:18:28.282944 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.282967 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 11:18:28.283346 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 -> Peer eb9fefb9d37a451eba23224c55ab3f82 (127.0.98.60:35285): Couldn't send request to peer eb9fefb9d37a451eba23224c55ab3f82. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35285: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 11:18:28.283679 7148 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:28.284021 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b6fdfc1f39714f61925ab759c7bfe515 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } } }
I20260927 11:18:28.284987 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } } }
I20260927 11:18:28.285102 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:28.285181 7148 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 11:18:28.285168 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } } }
I20260927 11:18:28.285218 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:28.285236 7148 log.cc:1196] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 11:18:28.285455 7148 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: Deleting consensus metadata
I20260927 11:18:28.285656 7148 master_runner.cc:315] Copying system tablet from 127.0.98.62:42973
I20260927 11:18:28.285666 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b6fdfc1f39714f61925ab759c7bfe515 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } } }
I20260927 11:18:28.286270 7148 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42973
I20260927 11:18:28.286605 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } } }
I20260927 11:18:28.286688 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:18:28.293023 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.286636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.0.98.62:42973)
0927 11:18:28.286674 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.286676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.287169 (+ 493us) server_negotiation.cc:407] Connection header received
0927 11:18:28.287242 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.287243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.287255 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.287304 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.287611 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.288113 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.288846 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.289220 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.289309 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.290304 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.290305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.290306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.290313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.291917 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.292242 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.292243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.292244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.292308 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.292518 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.292519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.292520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.292634 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.292710 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.292845 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.292923 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:18:28.293349 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.286717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:35664)
0927 11:18:28.287096 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.287107 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.287183 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.287308 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.287309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.287495 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.287534 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.287535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.288209 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.288210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.288754 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.288756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.289310 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.289311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.289371 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.289876 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.289881 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.290195 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.291988 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.291989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.291989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.292168 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.292374 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.292375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.292375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.292432 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.292700 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.292701 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.292747 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.292820 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.292923 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":43,"threads_started":1}
I20260927 11:18:28.293769 6514 tablet_copy_service.cc:140] P 96116c3dd29542429df49b14bbbca4a9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer eb9fefb9d37a451eba23224c55ab3f82 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35664)
I20260927 11:18:28.293869 6514 tablet_copy_service.cc:161] P 96116c3dd29542429df49b14bbbca4a9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer eb9fefb9d37a451eba23224c55ab3f82 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35664: session id = eb9fefb9d37a451eba23224c55ab3f82-00000000000000000000000000000000
I20260927 11:18:28.294449 6514 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 11:18:28.294929 7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 11:18:28.296027 7148 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: tablet copy: Starting download of 0 data blocks...
I20260927 11:18:28.296188 7148 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: tablet copy: Starting download of 1 WAL segments...
I20260927 11:18:28.296845 7148 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 11:18:28.297652 6514 tablet_copy_service.cc:342] P 96116c3dd29542429df49b14bbbca4a9: Request end of tablet copy session eb9fefb9d37a451eba23224c55ab3f82-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35664
I20260927 11:18:28.297706 6514 tablet_copy_service.cc:433] P 96116c3dd29542429df49b14bbbca4a9: ending tablet copy session eb9fefb9d37a451eba23224c55ab3f82-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer eb9fefb9d37a451eba23224c55ab3f82
I20260927 11:18:28.299985 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:28.301232 7254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:28.301455 7257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:28.301919 7255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:28.306017 7148 server_base.cc:1034] running on GCE node
I20260927 11:18:28.306129 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:28.306262 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:28.307389 7148 hybrid_clock.cc:648] HybridClock initialized: now 1790507908307384 us; error 30 us; skew 500 ppm
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:28.308810 7148 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 11:18:28.309096 7148 webserver.cc:460] Webserver started at http://127.0.98.60:41403/ using document root <none> and password file <none>
I20260927 11:18:28.309218 7148 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:28.309257 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:28.309902 7148 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:28.310408 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.310534 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.310571 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/wal
uuid: "eb9fefb9d37a451eba23224c55ab3f82"
format_stamp: "Formatted at 2026-09-27 11:18:28 on dist-test-slave-lsvg"
server_key: "e20e65e9fca3c50f8a30dca6691d3c9a"
server_key_iv: "2e73063d39e731b3e0d99959269ba392"
server_key_version: "encryptionkey@0"
I20260927 11:18:28.310632 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:28.344575 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:28.346457 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:28.349892 7148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35285
I20260927 11:18:28.349915 7316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35285 every 8 connection(s)
I20260927 11:18:28.350231 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/master-2/data/info.pb
I20260927 11:18:28.353649 7317 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:18:28.354123 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: Bootstrap starting.
I20260927 11:18:28.356343 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:18:28.356593 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: Bootstrap complete.
I20260927 11:18:28.356822 7317 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } }
I20260927 11:18:28.356912 7317 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb9fefb9d37a451eba23224c55ab3f82, State: Initialized, Role: LEARNER
I20260927 11:18:28.356961 7317 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } }
I20260927 11:18:28.357162 7317 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:28.357154 7320 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } } }
I20260927 11:18:28.357364 7320 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: This master's current role is: LEARNER
I20260927 11:18:28.358681 7331 catalog_manager.cc:1259] Loaded cluster ID: 8089438b1e9f4e119347432a98fdac7d
I20260927 11:18:28.358770 7331 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: loading cluster ID for follower catalog manager: success
I20260927 11:18:28.360069 7331 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: acquiring CA information for follower catalog manager: success
I20260927 11:18:28.360353 7331 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 11:18:28.370949 7241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:28.371243 7241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:28.371281 7241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:28.372988 7241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:28.373076 7241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:28.373123 7241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:28.375226 7241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42915
--tserver_master_addrs=127.0.98.62:42973,127.0.98.61:33667,127.0.98.60:35285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:28.375473 7241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:28.375707 7241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:28.380892 7337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:28.380906 7336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:28.380975 7339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:28.381398 7241 server_base.cc:1034] running on GCE node
I20260927 11:18:28.381661 7241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:28.381924 7241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:28.383102 7241 hybrid_clock.cc:648] HybridClock initialized: now 1790507908383088 us; error 28 us; skew 500 ppm
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:28.386462 7241 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 11:18:28.386996 7241 webserver.cc:460] Webserver started at http://127.0.98.1:42915/ using document root <none> and password file <none>
I20260927 11:18:28.387247 7241 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:28.387306 7241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:28.388675 7241 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:28.389312 7346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.389508 7241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:28.389606 7241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "fd95fd6097a444ac9235305e5ab60606"
format_stamp: "Formatted at 2026-09-27 11:18:26 on dist-test-slave-lsvg"
server_key: "7a0017794024a718047c2ef50d629edc"
server_key_iv: "3cb5b1e4465f54ee76dd9576df46fd1f"
server_key_version: "encryptionkey@0"
I20260927 11:18:28.389966 7241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:28.401342 7241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:28.403556 7241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:28.403710 7241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:28.403934 7241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:28.404229 7241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:28.404282 7241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.404322 7241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:28.404350 7241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.410727 7241 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33883
I20260927 11:18:28.410796 7459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33883 every 8 connection(s)
I20260927 11:18:28.411140 7241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:18:28.417454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 7241
I20260927 11:18:28.417594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 6690
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 11:18:28.423022 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.411881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46353 (local address 127.0.98.62:42973)
0927 11:18:28.412003 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.412006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.412254 (+ 248us) server_negotiation.cc:407] Connection header received
0927 11:18:28.413424 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.413426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.413444 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.413518 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.414278 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.415037 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.415923 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.416153 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.416219 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.419723 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.419725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.419726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.419735 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.421452 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.422140 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.422141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.422142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.422192 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.422403 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.422404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.422404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.422531 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.422571 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.422857 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.422934 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:18:28.423197 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.412335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35285 (local address 127.0.98.1:36805)
0927 11:18:28.412453 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.412461 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.413336 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.413644 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.413644 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.413992 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.414169 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.414176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.415348 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.415349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.416102 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.416103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.416551 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.416552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.416620 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.417231 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.417235 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.419021 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.421678 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.421678 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.421679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.421882 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.422122 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.422123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.422123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.422252 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.422535 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.422541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.422699 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.422815 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.422969 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":27,"thread_start_us":68,"threads_started":1}
I20260927 11:18:28.423326 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.411892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.1:46353)
0927 11:18:28.412124 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.412168 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.413320 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.413584 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.413588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.414005 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.414170 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.414176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.415127 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.415127 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.415836 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.415838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.416231 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.416231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.416384 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.416977 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.416984 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.419584 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.421545 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.421551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.421554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.422059 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.422263 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.422264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.422264 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.422331 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.422613 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.422613 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.422699 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.423133 (+ 434us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.423217 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":9,"thread_start_us":145,"threads_started":1}
I20260927 11:18:28.423718 7461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42973
I20260927 11:18:28.423815 7461 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:28.424008 7461 heartbeater.cc:511] Master 127.0.98.62:42973 requested a full tablet report, sending...
I20260927 11:18:28.424213 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.412530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36805 (local address 127.0.98.60:35285)
0927 11:18:28.412637 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.412638 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.412645 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:18:28.413505 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.413506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.413519 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.413604 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.414275 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.415265 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.416178 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.416465 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.416540 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.419133 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.419134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.419134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.419141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.421582 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.421972 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.421973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.421974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.422042 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.422327 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.422328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.422328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.422462 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.424004 (+ 1542us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.424044 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.424120 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":64,"threads_started":1}
I20260927 11:18:28.424490 7460 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35285
I20260927 11:18:28.424537 7460 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:28.424598 7460 heartbeater.cc:511] Master 127.0.98.60:35285 requested a full tablet report, sending...
I20260927 11:18:28.424834 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.411989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49373 (local address 127.0.98.61:33667)
0927 11:18:28.412038 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.412040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.412319 (+ 279us) server_negotiation.cc:407] Connection header received
0927 11:18:28.413505 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.413507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.413519 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.413572 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.414261 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.414752 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.415929 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.416235 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.416307 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.420344 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.420346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.420346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.420354 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.422625 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.423688 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.423689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.423690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.423749 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.423973 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.423974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.423975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.424141 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.424208 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.424663 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.424753 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:18:28.424966 7273 ts_manager.cc:194] Registered new tserver with Master: fd95fd6097a444ac9235305e5ab60606 (127.0.98.1:33883)
I20260927 11:18:28.425168 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.411892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33667 (local address 127.0.98.1:49373)
0927 11:18:28.412261 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.412271 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.413342 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.413631 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.413632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.413978 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.414169 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.414173 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.414856 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.414856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.415838 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.415840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.416308 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.416309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.416384 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.417005 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.417010 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.420255 (+ 3245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.423381 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.423381 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.423382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.423608 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.423831 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.423831 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.423832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.423889 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.424492 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.424493 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.424540 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.424632 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.424747 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":2,"thread_start_us":313,"threads_started":1}
I20260927 11:18:28.425181 6494 ts_manager.cc:194] Re-registered known tserver with Master: fd95fd6097a444ac9235305e5ab60606 (127.0.98.1:33883)
I20260927 11:18:28.425585 7462 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33667
I20260927 11:18:28.425633 7462 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:28.425652 6494 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46353
I20260927 11:18:28.425691 7462 heartbeater.cc:511] Master 127.0.98.61:33667 requested a full tablet report, sending...
I20260927 11:18:28.426011 7093 ts_manager.cc:194] Registered new tserver with Master: fd95fd6097a444ac9235305e5ab60606 (127.0.98.1:33883)
I20260927 11:18:28.426345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39715
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42973
--webserver_port=45151
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42973,127.0.98.61:33667
--tserver_master_addrs=127.0.98.62:42973,127.0.98.61:33667,127.0.98.60:35285 with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:28.507548 7468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:28.507750 7468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:28.507772 7468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:28.508823 7468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:28.508857 7468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:28.508881 7468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:28.510457 7468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45151
--tserver_master_addrs=127.0.98.62:42973,127.0.98.61:33667,127.0.98.60:35285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:28.510704 7468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:28.510967 7468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:28.516273 7473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:28.516273 7474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:28.516323 7476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:28.516484 7468 server_base.cc:1034] running on GCE node
I20260927 11:18:28.516743 7468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:28.516993 7468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:28.518194 7468 hybrid_clock.cc:648] HybridClock initialized: now 1790507908518172 us; error 39 us; skew 500 ppm
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:28.520642 7468 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 11:18:28.521116 7468 webserver.cc:460] Webserver started at http://127.0.98.2:45151/ using document root <none> and password file <none>
I20260927 11:18:28.521349 7468 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:28.521423 7468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:28.522859 7468 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:28.523571 7483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.523764 7468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:28.523833 7468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "bf9bc68d3c124f1da0f058f791c3a21f"
format_stamp: "Formatted at 2026-09-27 11:18:26 on dist-test-slave-lsvg"
server_key: "46a20c644c2895ac58382227ec2654f0"
server_key_iv: "9342397d3992f49599bdedf9afce52d1"
server_key_version: "encryptionkey@0"
I20260927 11:18:28.524137 7468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:28.541805 7468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:28.543663 7468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:28.543773 7468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:28.543967 7468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:28.544277 7468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:28.544312 7468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.544364 7468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:28.544389 7468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.549981 7468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39715
I20260927 11:18:28.550024 7596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39715 every 8 connection(s)
I20260927 11:18:28.550391 7468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:18:28.550767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 7468
I20260927 11:18:28.550922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 6824
I20260927 11:18:28.557536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40497
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42973
--webserver_port=37553
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42973,127.0.98.61:33667
--tserver_master_addrs=127.0.98.62:42973,127.0.98.61:33667,127.0.98.60:35285 with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 11:18:28.563908 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.551610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50199 (local address 127.0.98.60:35285)
0927 11:18:28.551653 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.551656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.551817 (+ 161us) server_negotiation.cc:407] Connection header received
0927 11:18:28.552553 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.552554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.552568 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.552621 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.553550 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.554320 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.555901 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.556216 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.556258 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.560230 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.560232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.560232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.560241 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.561929 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.562752 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.562753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.562753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.562818 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.563134 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.563135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.563135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.563483 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.563543 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.563727 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.563814 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:18:28.564378 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.551475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49771 (local address 127.0.98.62:42973)
0927 11:18:28.551520 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.551522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.551835 (+ 313us) server_negotiation.cc:407] Connection header received
0927 11:18:28.552401 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.552403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.552416 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.552496 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.553551 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.554331 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.555901 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.556216 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.556258 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.559475 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.559477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.559478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.559487 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.561671 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.562746 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.562747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.562747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.562818 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.563129 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.563129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.563130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.563272 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.563353 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.564219 (+ 866us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.564291 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:18:28.564625 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.556073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36711 (local address 127.0.98.61:33667)
0927 11:18:28.556138 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.556140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.556150 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:28.556173 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.556174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.556185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.556236 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.556632 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.557433 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.558842 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.559158 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.559239 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.561524 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.561525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.561525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.561532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.563256 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.563708 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.563709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.563709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.563772 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.564004 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.564005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.564006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.564148 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.564192 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.564424 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.564514 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 11:18:28.564723 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.551408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.2:49771)
0927 11:18:28.551660 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.551711 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.552293 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.552552 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.552555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.553176 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.553407 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.553413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.554465 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.554467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.555794 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.555796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.556352 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.556353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.556508 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.557358 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.557363 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.559362 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.562018 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.562025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.562027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.562661 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.562891 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.562892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.562892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.563053 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.563356 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.563362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.563526 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.563661 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.563825 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":45,"thread_start_us":69,"threads_started":1}
I20260927 11:18:28.564913 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.551122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35285 (local address 127.0.98.2:50199)
0927 11:18:28.551660 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.551711 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.552295 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.552643 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.552644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.553120 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.553407 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.553413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.554655 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.554656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.555794 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.555796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.556351 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.556352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.556508 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.557211 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.557221 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.560121 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.562020 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.562025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.562027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.562661 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.562893 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.562894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.562894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.563053 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.563570 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.563570 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.563617 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.564045 (+ 428us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.564141 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":13,"thread_start_us":78,"threads_started":1}
I20260927 11:18:28.565106 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.554526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33667 (local address 127.0.98.2:36711)
0927 11:18:28.554643 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.554654 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.554706 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.556282 (+ 1576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.556284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.556481 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.556546 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.556547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.557837 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.557838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.558750 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.558752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.559242 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.559243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.559314 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.559723 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.559728 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.561427 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.563356 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.563356 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.563357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.563613 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.563849 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.563850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.563851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.563912 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.564243 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.564244 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.564289 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.564367 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.564481 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260927 11:18:28.565188 7598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42973
I20260927 11:18:28.565276 7598 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:28.565479 7598 heartbeater.cc:511] Master 127.0.98.62:42973 requested a full tablet report, sending...
I20260927 11:18:28.565649 7599 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33667
I20260927 11:18:28.565704 7599 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:28.565778 7599 heartbeater.cc:511] Master 127.0.98.61:33667 requested a full tablet report, sending...
I20260927 11:18:28.565799 7597 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35285
I20260927 11:18:28.565832 7597 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:28.565914 7597 heartbeater.cc:511] Master 127.0.98.60:35285 requested a full tablet report, sending...
I20260927 11:18:28.566159 7093 ts_manager.cc:194] Registered new tserver with Master: bf9bc68d3c124f1da0f058f791c3a21f (127.0.98.2:39715)
I20260927 11:18:28.566375 6494 ts_manager.cc:194] Re-registered known tserver with Master: bf9bc68d3c124f1da0f058f791c3a21f (127.0.98.2:39715)
I20260927 11:18:28.566598 7273 ts_manager.cc:194] Registered new tserver with Master: bf9bc68d3c124f1da0f058f791c3a21f (127.0.98.2:39715)
I20260927 11:18:28.566859 6494 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49771
W20260927 11:18:28.645990 7604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:28.646232 7604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:28.646274 7604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:28.647414 7604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:28.647483 7604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:28.647528 7604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:28.649008 7604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37553
--tserver_master_addrs=127.0.98.62:42973,127.0.98.61:33667,127.0.98.60:35285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:28.649317 7604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:28.649564 7604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:28.654603 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:28.654644 7604 server_base.cc:1034] running on GCE node
W20260927 11:18:28.654592 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:28.654592 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:28.655032 7604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:28.655292 7604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:28.656451 7604 hybrid_clock.cc:648] HybridClock initialized: now 1790507908656447 us; error 37 us; skew 500 ppm
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:28.659107 7604 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 11:18:28.659546 7604 webserver.cc:460] Webserver started at http://127.0.98.3:37553/ using document root <none> and password file <none>
I20260927 11:18:28.659780 7604 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:28.659862 7604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:28.661250 7604 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:28.661821 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.661979 7604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.662083 7604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "4a8c2a0a48624715b798b132579a3ceb"
format_stamp: "Formatted at 2026-09-27 11:18:27 on dist-test-slave-lsvg"
server_key: "fd81d697f9da42f0c086280ca7c9b562"
server_key_iv: "53532e23d55bf46a197b76fe8705a886"
server_key_version: "encryptionkey@0"
I20260927 11:18:28.662426 7604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:28.694440 7604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:28.696502 7604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:28.696614 7604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:28.696784 7604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:28.697027 7604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:28.697121 7604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.697148 7604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:28.697161 7604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:28.702618 7604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40497
I20260927 11:18:28.702718 7732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40497 every 8 connection(s)
I20260927 11:18:28.703029 7604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:28.704393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 7604
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 11:18:28 dist-test-slave-lsvg krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507908, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 11:18:28.716180 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.703730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50837 (local address 127.0.98.62:42973)
0927 11:18:28.703774 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.703776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.704078 (+ 302us) server_negotiation.cc:407] Connection header received
0927 11:18:28.705704 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.705705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.705717 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.706140 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.706958 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.707750 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.709250 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.709546 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.709592 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.712640 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.712641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.712642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.712648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.714661 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.715387 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.715388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.715388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.715440 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.715642 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.715643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.715643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.715758 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.715796 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.716016 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.716088 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:18:28.716179 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.704222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51743 (local address 127.0.98.61:33667)
0927 11:18:28.704324 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.704325 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.704332 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:18:28.705514 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.705515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.705527 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.705582 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.706945 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.707715 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.709238 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.709515 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.709570 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.712639 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.712640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.712641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.712647 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.714497 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.715027 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.715028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.715028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.715080 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.715365 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.715366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.715367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.715523 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.715592 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.715975 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.716059 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 11:18:28.716782 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.704107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33667 (local address 127.0.98.3:51743)
0927 11:18:28.704227 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.704236 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.705389 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.706150 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.706151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.706532 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.706833 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.706838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.707829 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.707830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.709161 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.709163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.709622 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.709623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.709751 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.710344 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.710354 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.712535 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.714588 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.714594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.714596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.714948 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.715152 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.715152 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.715153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.715274 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.715624 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.715631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.715801 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.716316 (+ 515us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.716496 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":41,"thread_start_us":62,"threads_started":1}
I20260927 11:18:28.717352 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.61:33667
I20260927 11:18:28.717458 7735 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:28.717649 7735 heartbeater.cc:511] Master 127.0.98.61:33667 requested a full tablet report, sending...
I20260927 11:18:28.718062 7093 ts_manager.cc:194] Registered new tserver with Master: 4a8c2a0a48624715b798b132579a3ceb (127.0.98.3:40497)
I20260927 11:18:28.718631 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.703745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.3:50837)
0927 11:18:28.703934 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.703985 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.705626 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.706214 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.706215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.706536 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.706833 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.706840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.707840 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.707841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.709157 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.709160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.709673 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.709674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.709755 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.710563 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.710569 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.712524 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.715108 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.715109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.715109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.715302 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.715515 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.715516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.715516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.715572 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.715857 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.715857 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.715896 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.717298 (+ 1402us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.718525 (+ 1227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":48,"thread_start_us":95,"threads_started":1}
I20260927 11:18:28.718823 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.703718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49813 (local address 127.0.98.60:35285)
0927 11:18:28.703768 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.703770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.704094 (+ 324us) server_negotiation.cc:407] Connection header received
0927 11:18:28.705672 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.705673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.705686 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.705745 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.710098 (+ 4353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.710793 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.711673 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.711931 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.712021 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.715090 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.715091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.715091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.715098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.716877 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.717856 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.717857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.717858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.717914 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.718109 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.718110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.718110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.718240 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.718319 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.718513 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.718743 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:18:28.719051 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.705642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55172 (local address 127.0.98.61:33667)
0927 11:18:28.706361 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.706363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.706369 (+ 6us) server_negotiation.cc:407] Connection header received
0927 11:18:28.706388 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.706388 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.706398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.706505 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.707204 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.707896 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.708931 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.709266 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.709320 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.714334 (+ 5014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.714336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.714337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.714345 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.717736 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.718147 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.718148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.718148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.718203 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.718400 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.718401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.718402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.718523 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.718697 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.718880 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.718957 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":51,"threads_started":1}
I20260927 11:18:28.719414 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.703745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35285 (local address 127.0.98.3:49813)
0927 11:18:28.703934 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.703985 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.705570 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.705973 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.705979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.709975 (+ 3996us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:28.710033 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.710035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.710897 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.710898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.711574 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.711577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.712025 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.712026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.712087 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.712850 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:28.712874 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:28.714989 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:28.717351 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.717352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.717352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.717768 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.717983 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:28.717983 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:28.717984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:28.718038 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:28.718343 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:28.718344 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:28.718385 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.718929 (+ 544us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.719300 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"spinlock_wait_cycles":8448,"thread_start_us":89,"threads_started":1}
I20260927 11:18:28.719452 7734 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42973
I20260927 11:18:28.719491 7734 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:28.719555 7734 heartbeater.cc:511] Master 127.0.98.62:42973 requested a full tablet report, sending...
I20260927 11:18:28.719722 7733 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35285
I20260927 11:18:28.719779 7733 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:28.719774 6494 ts_manager.cc:194] Re-registered known tserver with Master: 4a8c2a0a48624715b798b132579a3ceb (127.0.98.3:40497)
I20260927 11:18:28.719847 7733 heartbeater.cc:511] Master 127.0.98.60:35285 requested a full tablet report, sending...
I20260927 11:18:28.720096 7273 ts_manager.cc:194] Registered new tserver with Master: 4a8c2a0a48624715b798b132579a3ceb (127.0.98.3:40497)
I20260927 11:18:28.720286 6494 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50837
I20260927 11:18:28.721073 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.705956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41086 (local address 127.0.98.60:35285)
0927 11:18:28.706174 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.706175 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.706710 (+ 535us) server_negotiation.cc:407] Connection header received
0927 11:18:28.706737 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.706738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.706749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.706799 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.707417 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.708131 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.709186 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.709493 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.709532 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.716508 (+ 6976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.716509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.716509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.716516 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.718506 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.719852 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.719853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.719854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.720265 (+ 411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.720438 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.720439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.720440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.720555 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.720661 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.720762 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.720842 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":43,"threads_started":1}
I20260927 11:18:28.722945 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.705543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.0.98.62:42973)
0927 11:18:28.710152 (+ 4609us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.710154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.710161 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:18:28.710181 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.710182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.710194 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.710253 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.712898 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.714867 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.716626 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.717204 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.717263 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.720644 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.720646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.720646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.720652 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.722099 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.722401 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.722401 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.722402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.722457 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.722596 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.722596 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.722596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.722676 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.722731 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.722797 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.722852 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4580,"thread_start_us":79,"threads_started":1}
I20260927 11:18:28.724462 6494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:28.726188 6494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:28.733639 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.729827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39715 (local address 127.0.0.1:40130)
0927 11:18:28.729872 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.729883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.729920 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.730514 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.730515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.730517 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:28.730560 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.730562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.731568 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.731569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.732474 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.732476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.733388 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.733389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.733451 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.733455 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.733535 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.733570 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 11:18:28.733675 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.730128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.0.98.2:39715)
0927 11:18:28.730327 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.730331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.730341 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:28.730385 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.730388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.730404 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.730639 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:28.730702 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.731477 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.732558 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.733313 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.733381 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.733456 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.733553 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.733595 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260927 11:18:28.734048 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.730026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40497 (local address 127.0.0.1:40786)
0927 11:18:28.730618 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.730628 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.730678 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.730853 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.730854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.730856 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:28.730888 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.730889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.732024 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.732025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.732893 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.732895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.733808 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.733809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.733878 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.733882 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.733943 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.733978 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":75,"threads_started":1}
I20260927 11:18:28.734246 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.730113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.0.98.3:40497)
0927 11:18:28.730240 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.730244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.730679 (+ 435us) server_negotiation.cc:407] Connection header received
0927 11:18:28.730746 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.730749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.730764 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.730843 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:28.730952 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.731937 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.732986 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.733638 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.733703 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.733778 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.733968 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.734013 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260927 11:18:28.734462 7531 tablet_service.cc:1467] Processing CreateTablet for tablet e35c5bd74c6c4d7d9b0e318f33c7a18d (DEFAULT_TABLE table=test-table [id=d5e67d47f7a443c4a62d229dc4594c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:28.734845 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e35c5bd74c6c4d7d9b0e318f33c7a18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:28.734848 7667 tablet_service.cc:1467] Processing CreateTablet for tablet e35c5bd74c6c4d7d9b0e318f33c7a18d (DEFAULT_TABLE table=test-table [id=d5e67d47f7a443c4a62d229dc4594c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:28.735154 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e35c5bd74c6c4d7d9b0e318f33c7a18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:28.737800 7760 tablet_bootstrap.cc:492] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb: Bootstrap starting.
I20260927 11:18:28.737998 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.729835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33883 (local address 127.0.0.1:32864)
0927 11:18:28.729996 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.730007 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.730060 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.730510 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.730511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.730514 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:28.730564 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.730565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.731888 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.731889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.733204 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.733207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.737585 (+ 4378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.737586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.737657 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.737661 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.737841 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.737885 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":13,"thread_start_us":95,"threads_started":1}
I20260927 11:18:28.738013 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.729947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.0.98.1:33883)
0927 11:18:28.730365 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.730368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.730378 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:28.730411 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.730413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.730439 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.730513 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:28.730646 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.731796 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.733300 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.737494 (+ 4194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.737583 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.737655 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.737883 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.737936 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":91,"threads_started":1}
I20260927 11:18:28.737998 7759 tablet_bootstrap.cc:492] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f: Bootstrap starting.
I20260927 11:18:28.738682 7760 tablet_bootstrap.cc:654] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:28.738694 7759 tablet_bootstrap.cc:654] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:28.739145 7394 tablet_service.cc:1467] Processing CreateTablet for tablet e35c5bd74c6c4d7d9b0e318f33c7a18d (DEFAULT_TABLE table=test-table [id=d5e67d47f7a443c4a62d229dc4594c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:28.739394 7759 log.cc:824] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:28.739416 7760 log.cc:824] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:28.739496 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e35c5bd74c6c4d7d9b0e318f33c7a18d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:28.740558 7759 tablet_bootstrap.cc:492] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f: No bootstrap required, opened a new log
I20260927 11:18:28.740633 7759 ts_tablet_manager.cc:1397] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 11:18:28.741380 7760 tablet_bootstrap.cc:492] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb: No bootstrap required, opened a new log
I20260927 11:18:28.741467 7760 ts_tablet_manager.cc:1397] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 11:18:28.742833 7760 raft_consensus.cc:348] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } }
I20260927 11:18:28.742990 7760 raft_consensus.cc:374] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:28.743031 7760 raft_consensus.cc:729] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a8c2a0a48624715b798b132579a3ceb, State: Initialized, Role: FOLLOWER
I20260927 11:18:28.743160 7760 consensus_queue.cc:260] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } }
I20260927 11:18:28.743160 7759 raft_consensus.cc:348] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } }
I20260927 11:18:28.743315 7759 raft_consensus.cc:374] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:28.743358 7759 raft_consensus.cc:729] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9bc68d3c124f1da0f058f791c3a21f, State: Initialized, Role: FOLLOWER
I20260927 11:18:28.743423 7760 ts_tablet_manager.cc:1428] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:18:28.743402 7763 tablet_bootstrap.cc:492] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606: Bootstrap starting.
I20260927 11:18:28.743475 7759 consensus_queue.cc:260] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } }
I20260927 11:18:28.743563 7734 heartbeater.cc:503] Master 127.0.98.62:42973 was elected leader, sending a full tablet report...
I20260927 11:18:28.743714 7759 ts_tablet_manager.cc:1428] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:18:28.744118 7598 heartbeater.cc:503] Master 127.0.98.62:42973 was elected leader, sending a full tablet report...
I20260927 11:18:28.744571 7763 tablet_bootstrap.cc:654] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:28.744962 7763 log.cc:824] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:28.745927 7763 tablet_bootstrap.cc:492] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606: No bootstrap required, opened a new log
I20260927 11:18:28.745982 7763 ts_tablet_manager.cc:1397] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 11:18:28.748035 7763 raft_consensus.cc:348] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } }
I20260927 11:18:28.748157 7763 raft_consensus.cc:374] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:28.748178 7763 raft_consensus.cc:729] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95fd6097a444ac9235305e5ab60606, State: Initialized, Role: FOLLOWER
I20260927 11:18:28.748313 7763 consensus_queue.cc:260] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } }
I20260927 11:18:28.748555 7763 ts_tablet_manager.cc:1428] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:18:28.748769 7461 heartbeater.cc:503] Master 127.0.98.62:42973 was elected leader, sending a full tablet report...
W20260927 11:18:28.802175 7600 tablet.cc:2369] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:28.809576 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.806896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35285 (local address 127.0.0.1:41092)
0927 11:18:28.806981 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.806993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.807064 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.807245 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.807246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.807247 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:28.807310 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.807312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.807994 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.807996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.808718 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.808720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.809321 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.809322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.809378 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.809382 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.809444 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.809498 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 11:18:28.809573 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.806999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.0.98.60:35285)
0927 11:18:28.807058 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.807060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.807070 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:28.807138 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.807139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.807166 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.807242 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:28.807390 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.807902 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.808809 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.809236 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.809322 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.809404 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.809461 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.809505 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:18:28.810014 7291 raft_consensus.cc:1203] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260927 11:18:28.810964 7320 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96116c3dd29542429df49b14bbbca4a9. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } } }
I20260927 11:18:28.811071 7320 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: This master's current role is: LEARNER
I20260927 11:18:28.839629 7765 raft_consensus.cc:482] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:28.839792 7765 raft_consensus.cc:504] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } }
I20260927 11:18:28.840207 7765 leader_election.cc:288] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd95fd6097a444ac9235305e5ab60606 (127.0.98.1:33883), 4a8c2a0a48624715b798b132579a3ceb (127.0.98.3:40497)
I20260927 11:18:28.843189 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.840602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48751 (local address 127.0.98.3:40497)
0927 11:18:28.840657 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.840659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.840669 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:28.840693 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.840694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.840712 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.840772 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:28.840946 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.841422 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.842419 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.842819 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.842907 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.842916 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.843086 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.843124 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 11:18:28.843190 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.840484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52467 (local address 127.0.98.1:33883)
0927 11:18:28.840531 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.840534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.840548 (+ 14us) server_negotiation.cc:407] Connection header received
0927 11:18:28.840607 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.840608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.840621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.840718 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:28.840882 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.841479 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.842418 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.842809 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.842904 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.842915 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.843068 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.843106 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:18:28.843211 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.840411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33883 (local address 127.0.98.2:52467)
0927 11:18:28.840476 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.840488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.840538 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.840724 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.840725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.840726 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:28.840795 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.840797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.841563 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.841564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.842323 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.842325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.842903 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.842904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.842982 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.842986 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.843062 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.843125 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":24}
I20260927 11:18:28.843199 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.840537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40497 (local address 127.0.98.2:48751)
0927 11:18:28.840575 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:28.840584 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:28.840614 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:28.840789 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:28.840790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:28.840794 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:28.840854 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.840856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.841525 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.841526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.842323 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:28.842325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.842905 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.842906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:28.842998 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.843001 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:28.843076 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 11:18:28.843119 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 11:18:28.843578 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e35c5bd74c6c4d7d9b0e318f33c7a18d" candidate_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95fd6097a444ac9235305e5ab60606" is_pre_election: true
I20260927 11:18:28.843740 7414 raft_consensus.cc:2454] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9bc68d3c124f1da0f058f791c3a21f in term 0.
I20260927 11:18:28.843827 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e35c5bd74c6c4d7d9b0e318f33c7a18d" candidate_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a8c2a0a48624715b798b132579a3ceb" is_pre_election: true
I20260927 11:18:28.844000 7687 raft_consensus.cc:2454] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9bc68d3c124f1da0f058f791c3a21f in term 0.
I20260927 11:18:28.844004 7487 leader_election.cc:302] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf9bc68d3c124f1da0f058f791c3a21f, fd95fd6097a444ac9235305e5ab60606; no voters:
I20260927 11:18:28.844129 7765 raft_consensus.cc:2781] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:28.844202 7765 raft_consensus.cc:482] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:28.844240 7765 raft_consensus.cc:3037] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:28.844944 7765 raft_consensus.cc:504] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } }
I20260927 11:18:28.845118 7765 leader_election.cc:288] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [CANDIDATE]: Term 1 election: Requested vote from peers fd95fd6097a444ac9235305e5ab60606 (127.0.98.1:33883), 4a8c2a0a48624715b798b132579a3ceb (127.0.98.3:40497)
I20260927 11:18:28.845294 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e35c5bd74c6c4d7d9b0e318f33c7a18d" candidate_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd95fd6097a444ac9235305e5ab60606"
I20260927 11:18:28.845383 7414 raft_consensus.cc:3037] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:28.845294 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e35c5bd74c6c4d7d9b0e318f33c7a18d" candidate_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a8c2a0a48624715b798b132579a3ceb"
I20260927 11:18:28.845872 7687 raft_consensus.cc:3037] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:28.846392 7414 raft_consensus.cc:2454] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9bc68d3c124f1da0f058f791c3a21f in term 1.
I20260927 11:18:28.846557 7487 leader_election.cc:302] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf9bc68d3c124f1da0f058f791c3a21f, fd95fd6097a444ac9235305e5ab60606; no voters:
I20260927 11:18:28.846685 7765 raft_consensus.cc:2781] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:28.846791 7687 raft_consensus.cc:2454] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9bc68d3c124f1da0f058f791c3a21f in term 1.
I20260927 11:18:28.846865 7765 raft_consensus.cc:686] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [term 1 LEADER]: Becoming Leader. State: Replica: bf9bc68d3c124f1da0f058f791c3a21f, State: Running, Role: LEADER
I20260927 11:18:28.846963 7765 consensus_queue.cc:237] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } }
I20260927 11:18:28.847783 6494 catalog_manager.cc:5495] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9bc68d3c124f1da0f058f791c3a21f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } health_report { overall_health: UNKNOWN } } }
I20260927 11:18:28.848448 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: true } } }
I20260927 11:18:28.848577 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:28.849149 7774 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9: attempting to promote NON_VOTER eb9fefb9d37a451eba23224c55ab3f82 to VOTER
I20260927 11:18:28.849277 7774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } }
I20260927 11:18:28.849831 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 FOLLOWER]: Refusing update from remote peer 96116c3dd29542429df49b14bbbca4a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 11:18:28.849836 7291 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 1 LEARNER]: Refusing update from remote peer 96116c3dd29542429df49b14bbbca4a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 11:18:28.850039 7245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 11:18:28.850133 7247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 11:18:28.850795 7774 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, eb9fefb9d37a451eba23224c55ab3f82 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } } }
I20260927 11:18:28.850960 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, eb9fefb9d37a451eba23224c55ab3f82 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } } }
I20260927 11:18:28.851763 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } } }
I20260927 11:18:28.851790 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } } }
I20260927 11:18:28.851856 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:28.851877 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:28.851938 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } } }
I20260927 11:18:28.852010 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:28.851987 7776 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } } }
I20260927 11:18:28.851974 7291 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, eb9fefb9d37a451eba23224c55ab3f82 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } } }
I20260927 11:18:28.852142 7776 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:18:28.852845 7320 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96116c3dd29542429df49b14bbbca4a9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } } }
I20260927 11:18:28.852936 7320 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:18:28.877086 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.874644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40134 (local address 127.0.98.2:39715)
0927 11:18:28.874694 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.874696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.874707 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:18:28.874754 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.874755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.874775 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.874904 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:28.875015 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.875487 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.876250 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.876465 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.876545 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.876720 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:28.876828 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:28.876926 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.876945 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.876965 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:18:28.878968 7687 raft_consensus.cc:1261] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb [term 1 FOLLOWER]: Refusing update from remote peer bf9bc68d3c124f1da0f058f791c3a21f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:28.878968 7414 raft_consensus.cc:1261] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606 [term 1 FOLLOWER]: Refusing update from remote peer bf9bc68d3c124f1da0f058f791c3a21f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:28.879328 7765 consensus_queue.cc:1035] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:28.879505 7772 consensus_queue.cc:1035] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:28.881805 7778 mvcc.cc:205] Tried to move back new op lower bound from 7333920394765471744 to 7333920394637774848. Current Snapshot: MvccSnapshot[applied={T|T < 7333920394765471744}]
I20260927 11:18:28.882441 7780 mvcc.cc:205] Tried to move back new op lower bound from 7333920394765471744 to 7333920394637774848. Current Snapshot: MvccSnapshot[applied={T|T < 7333920394765471744 or (T in {7333920394765471744})}]
I20260927 11:18:28.884850 6494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:28.884943 6494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:28.886638 6494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96116c3dd29542429df49b14bbbca4a9: Sending DeleteTablet for 3 replicas of tablet e35c5bd74c6c4d7d9b0e318f33c7a18d
I20260927 11:18:28.886891 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet e35c5bd74c6c4d7d9b0e318f33c7a18d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40130
I20260927 11:18:28.886945 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet e35c5bd74c6c4d7d9b0e318f33c7a18d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32864
I20260927 11:18:28.887131 7788 tablet_replica.cc:330] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f: stopping tablet replica
I20260927 11:18:28.887204 7789 tablet_replica.cc:330] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606: stopping tablet replica
I20260927 11:18:28.887238 7788 raft_consensus.cc:2229] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:28.887238 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet e35c5bd74c6c4d7d9b0e318f33c7a18d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40786
I20260927 11:18:28.887300 7789 raft_consensus.cc:2229] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:28.887341 7788 raft_consensus.cc:2258] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:28.887389 7789 raft_consensus.cc:2258] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:28.887415 7666 tablet_service.cc:1514] Processing DeleteTablet for tablet e35c5bd74c6c4d7d9b0e318f33c7a18d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40786
I20260927 11:18:28.887763 7788 ts_tablet_manager.cc:1905] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:28.887791 7789 ts_tablet_manager.cc:1905] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:28.889006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 6462
I20260927 11:18:28.889906 7788 ts_tablet_manager.cc:1918] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:28.889971 7788 log.cc:1196] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/e35c5bd74c6c4d7d9b0e318f33c7a18d
I20260927 11:18:28.890115 7789 ts_tablet_manager.cc:1918] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:28.890177 7789 log.cc:1196] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-0/wal/wals/e35c5bd74c6c4d7d9b0e318f33c7a18d
I20260927 11:18:28.890229 7788 ts_tablet_manager.cc:1939] T e35c5bd74c6c4d7d9b0e318f33c7a18d P bf9bc68d3c124f1da0f058f791c3a21f: Deleting consensus metadata
I20260927 11:18:28.890442 7789 ts_tablet_manager.cc:1939] T e35c5bd74c6c4d7d9b0e318f33c7a18d P fd95fd6097a444ac9235305e5ab60606: Deleting consensus metadata
I20260927 11:18:28.893550 7790 tablet_replica.cc:330] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb: stopping tablet replica
I20260927 11:18:28.893786 7790 raft_consensus.cc:2229] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:28.893975 7790 raft_consensus.cc:2258] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:28.894518 7790 ts_tablet_manager.cc:1905] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 11:18:28.895080 7347 connection.cc:535] server connection from 127.0.0.1:32864 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 11:18:28.895277 7485 connection.cc:535] server connection from 127.0.0.1:40130 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 11:18:28.896571 7790 ts_tablet_manager.cc:1918] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:28.896723 7790 log.cc:1196] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/e35c5bd74c6c4d7d9b0e318f33c7a18d
I20260927 11:18:28.897076 7790 ts_tablet_manager.cc:1939] T e35c5bd74c6c4d7d9b0e318f33c7a18d P 4a8c2a0a48624715b798b132579a3ceb: Deleting consensus metadata
W20260927 11:18:28.897534 7620 connection.cc:441] server connection from 127.0.0.1:40786 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40786 (request call id 1) could send its response
W20260927 11:18:28.897606 7620 connection.cc:441] server connection from 127.0.0.1:40786 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40786 (request call id 2) could send its response
I20260927 11:18:28.901165 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.893208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.0.98.60:35285)
0927 11:18:28.893307 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.893309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.893405 (+ 96us) server_negotiation.cc:407] Connection header received
0927 11:18:28.893461 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.893462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.893484 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.893546 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.894392 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.895220 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.896278 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.896590 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.896632 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.898297 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.898298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.898298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.898305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.899960 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.900340 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.900341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.900341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.900389 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.900589 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.900590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.900590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.900702 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.900770 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.900974 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.901069 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 11:18:28.905464 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.897484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.0.98.61:33667)
0927 11:18:28.897679 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.897681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.898003 (+ 322us) server_negotiation.cc:407] Connection header received
0927 11:18:28.898107 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.898108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.898123 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.898154 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.899166 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.899879 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.900817 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.901156 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.901193 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.902756 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.902758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.902758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.902766 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.904151 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.904535 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.904536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.904536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.904594 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.904938 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.904939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.904939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.905034 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.905167 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.905264 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.905347 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260927 11:18:28.905695 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:28.897994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.0.98.60:35285)
0927 11:18:28.898032 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:28.898034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:28.898202 (+ 168us) server_negotiation.cc:407] Connection header received
0927 11:18:28.898290 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:28.898291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:28.898300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:28.898331 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:28.898973 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.899701 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.900737 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:28.901025 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:28.901080 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:28.902625 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:28.902626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:28.902626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:28.902632 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:28.904431 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.904819 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.904820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.904821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.904869 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:28.905117 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:28.905118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:28.905118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:28.905236 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:28.905303 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:28.905509 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 11:18:28.905577 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:18:29.883339 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:29.882971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.1:45843)
0927 11:18:29.883140 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:29.883283 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260927 11:18:29.883407 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:29.883163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.2:38009)
0927 11:18:29.883289 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:29.883361 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
W20260927 11:18:29.883601 7598 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
W20260927 11:18:29.883708 7461 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
I20260927 11:18:29.887367 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:29.887170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.3:58723)
0927 11:18:29.887267 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:29.887331 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":62,"threads_started":1}
W20260927 11:18:29.887537 7734 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42973 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
I20260927 11:18:30.386715 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96116c3dd29542429df49b14bbbca4a9)
I20260927 11:18:30.386860 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } }
I20260927 11:18:30.387153 7802 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 96116c3dd29542429df49b14bbbca4a9)
I20260927 11:18:30.387185 7801 leader_election.cc:288] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96116c3dd29542429df49b14bbbca4a9 (127.0.98.62:42973), eb9fefb9d37a451eba23224c55ab3f82 (127.0.98.60:35285)
I20260927 11:18:30.387320 7802 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } }
I20260927 11:18:30.387485 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.387279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:36118)
0927 11:18:30.387394 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.387441 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":52,"threads_started":1}
W20260927 11:18:30.387598 7081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
I20260927 11:18:30.387957 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.387759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:36134)
0927 11:18:30.387870 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.387913 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
W20260927 11:18:30.388054 7268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
I20260927 11:18:30.388281 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.387988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:36136)
0927 11:18:30.388229 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.388258 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
W20260927 11:18:30.388361 7081 leader_election.cc:334] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96116c3dd29542429df49b14bbbca4a9 (127.0.98.62:42973): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
I20260927 11:18:30.388298 7802 leader_election.cc:288] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96116c3dd29542429df49b14bbbca4a9 (127.0.98.62:42973), b6fdfc1f39714f61925ab759c7bfe515 (127.0.98.61:33667)
I20260927 11:18:30.388653 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.388565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:36138)
0927 11:18:30.388601 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.388630 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 11:18:30.388782 7268 leader_election.cc:334] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 96116c3dd29542429df49b14bbbca4a9 (127.0.98.62:42973): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
I20260927 11:18:30.390739 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.387479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.0.98.60:35285)
0927 11:18:30.387646 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:30.387648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:30.387660 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:18:30.387720 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:30.387721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:30.387733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:30.387804 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:30.388005 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.388767 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.389864 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.390428 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.390506 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.390518 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:30.390669 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 11:18:30.390697 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":56,"threads_started":1}
I20260927 11:18:30.390776 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.387342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35285 (local address 127.0.0.1:41108)
0927 11:18:30.387589 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.387599 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:30.387657 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:30.387849 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:30.387850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:30.387852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:30.387911 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.387913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.388832 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.388833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.389775 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.389777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.390525 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.390526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.390592 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.390596 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:30.390665 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 11:18:30.390706 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":46,"threads_started":1}
I20260927 11:18:30.391057 7291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6fdfc1f39714f61925ab759c7bfe515" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb9fefb9d37a451eba23224c55ab3f82" is_pre_election: true
I20260927 11:18:30.391173 7291 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6fdfc1f39714f61925ab759c7bfe515 in term 1.
I20260927 11:18:30.391332 7078 leader_election.cc:302] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b6fdfc1f39714f61925ab759c7bfe515, eb9fefb9d37a451eba23224c55ab3f82; no voters: 96116c3dd29542429df49b14bbbca4a9
I20260927 11:18:30.391454 7801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 11:18:30.391503 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 96116c3dd29542429df49b14bbbca4a9)
I20260927 11:18:30.391507 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.387980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33667 (local address 127.0.0.1:55184)
0927 11:18:30.388107 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.388117 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:30.388165 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:30.388387 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:30.388388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:30.388390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:30.388438 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.388440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.389354 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.389355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.390425 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.390427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.391265 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.391266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.391330 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.391334 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:30.391392 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 11:18:30.391439 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":48,"threads_started":1}
I20260927 11:18:30.391533 7801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:18:30.392247 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } }
I20260927 11:18:30.392421 7801 leader_election.cc:288] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [CANDIDATE]: Term 2 election: Requested vote from peers 96116c3dd29542429df49b14bbbca4a9 (127.0.98.62:42973), eb9fefb9d37a451eba23224c55ab3f82 (127.0.98.60:35285)
I20260927 11:18:30.392551 7291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6fdfc1f39714f61925ab759c7bfe515" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb9fefb9d37a451eba23224c55ab3f82"
I20260927 11:18:30.392583 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.388104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.0.98.61:33667)
0927 11:18:30.388260 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:30.388262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:30.388268 (+ 6us) server_negotiation.cc:407] Connection header received
0927 11:18:30.388288 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:30.388289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:30.388298 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:30.388500 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:30.388533 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.389277 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.390522 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.391154 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.391221 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.391232 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:30.391416 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 11:18:30.392516 (+ 1100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260927 11:18:30.392637 7291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:18:30.392659 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.392512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:36146)
0927 11:18:30.392553 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.392578 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 11:18:30.392765 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb9fefb9d37a451eba23224c55ab3f82" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b6fdfc1f39714f61925ab759c7bfe515" is_pre_election: true
I20260927 11:18:30.392863 7104 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb9fefb9d37a451eba23224c55ab3f82 in current term 2: Already voted for candidate b6fdfc1f39714f61925ab759c7bfe515 in this term.
I20260927 11:18:30.392975 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.392865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:36162)
0927 11:18:30.392922 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.392951 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 11:18:30.393031 7081 leader_election.cc:334] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 96116c3dd29542429df49b14bbbca4a9 (127.0.98.62:42973): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
I20260927 11:18:30.393034 7265 leader_election.cc:302] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb9fefb9d37a451eba23224c55ab3f82; no voters: 96116c3dd29542429df49b14bbbca4a9, b6fdfc1f39714f61925ab759c7bfe515
I20260927 11:18:30.393489 7291 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6fdfc1f39714f61925ab759c7bfe515 in term 2.
I20260927 11:18:30.393564 7802 raft_consensus.cc:2726] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:18:30.393649 7078 leader_election.cc:302] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b6fdfc1f39714f61925ab759c7bfe515, eb9fefb9d37a451eba23224c55ab3f82; no voters: 96116c3dd29542429df49b14bbbca4a9
I20260927 11:18:30.393754 7801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 11:18:30.393888 7801 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [term 2 LEADER]: Becoming Leader. State: Replica: b6fdfc1f39714f61925ab759c7bfe515, State: Running, Role: LEADER
I20260927 11:18:30.393944 7801 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } }
I20260927 11:18:30.394304 7811 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6fdfc1f39714f61925ab759c7bfe515. Latest consensus state: current_term: 2 leader_uuid: "b6fdfc1f39714f61925ab759c7bfe515" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } } }
I20260927 11:18:30.394384 7811 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:30.394521 7813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:30.394974 7813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:30.395043 7813 catalog_manager.cc:1259] Loaded cluster ID: 8089438b1e9f4e119347432a98fdac7d
I20260927 11:18:30.395092 7813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:30.395252 7813 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:30.395334 7813 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: Loaded TSK: 0
I20260927 11:18:30.395514 7813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:18:30.437570 7093 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:30.439131 7093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:30.440177 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.440073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:36186)
0927 11:18:30.440119 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.440149 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 11:18:30.440203 7291 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [term 2 FOLLOWER]: Refusing update from remote peer b6fdfc1f39714f61925ab759c7bfe515: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 11:18:30.440518 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.440425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.0.1:36192)
0927 11:18:30.440470 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.440501 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 11:18:30.440459 7801 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 11:18:30.440623 7081 consensus_peers.cc:487] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 -> Peer 96116c3dd29542429df49b14bbbca4a9 (127.0.98.62:42973): Couldn't send request to peer 96116c3dd29542429df49b14bbbca4a9. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 11:18:30.441139 7802 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6fdfc1f39714f61925ab759c7bfe515. Latest consensus state: current_term: 2 leader_uuid: "b6fdfc1f39714f61925ab759c7bfe515" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } } }
I20260927 11:18:30.441252 7802 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:18:30.441643 7814 mvcc.cc:205] Tried to move back new op lower bound from 7333920401160839168 to 7333920400974372864. Current Snapshot: MvccSnapshot[applied={T|T < 7333920401160839168}]
I20260927 11:18:30.441720 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b6fdfc1f39714f61925ab759c7bfe515" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } } }
I20260927 11:18:30.441797 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:30.441797 7802 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b6fdfc1f39714f61925ab759c7bfe515" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "96116c3dd29542429df49b14bbbca4a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42973 } } peers { permanent_uuid: "b6fdfc1f39714f61925ab759c7bfe515" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33667 } attrs { promote: false } } peers { permanent_uuid: "eb9fefb9d37a451eba23224c55ab3f82" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35285 } attrs { promote: false } } }
I20260927 11:18:30.441900 7802 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9fefb9d37a451eba23224c55ab3f82 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:18:30.442000 7815 mvcc.cc:205] Tried to move back new op lower bound from 7333920401160839168 to 7333920400974372864. Current Snapshot: MvccSnapshot[applied={T|T < 7333920401160839168 or (T in {7333920401160839168})}]
I20260927 11:18:30.447209 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.443710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40497 (local address 127.0.0.1:40790)
0927 11:18:30.443763 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.443773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:30.443820 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:30.444478 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:30.444478 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:30.444479 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:30.444505 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.444506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.445332 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.445334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.446222 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.446224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.446957 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.446957 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.447024 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.447029 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:18:30.447084 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 11:18:30.447132 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 11:18:30.447219 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.443773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.0.98.3:40497)
0927 11:18:30.444374 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:30.444376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:30.444383 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:18:30.444407 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:30.444407 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:30.444417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:30.444479 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:30.444562 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.445253 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.446304 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.446867 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.446955 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.446969 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:30.447106 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 11:18:30.447149 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":72,"threads_started":1}
I20260927 11:18:30.447400 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.444095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.0.98.1:33883)
0927 11:18:30.444190 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:30.444192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:30.444199 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:18:30.444223 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:30.444223 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:30.444231 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:30.444292 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:30.444423 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.444963 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.446389 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.446962 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.447026 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.447038 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:30.447239 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 11:18:30.447290 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":46,"threads_started":1}
I20260927 11:18:30.447436 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.443806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33883 (local address 127.0.0.1:32870)
0927 11:18:30.443931 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.443940 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:30.443981 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:30.444307 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:30.444308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:30.444309 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:30.444341 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.444342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.445041 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.445042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.446318 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.446319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.447035 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.447036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.447097 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.447101 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:30.447146 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 11:18:30.447176 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":91,"thread_start_us":84,"threads_started":1}
I20260927 11:18:30.447672 7666 tablet_service.cc:1467] Processing CreateTablet for tablet bab08c1f7dda4ff8aa7968a5c93ee124 (DEFAULT_TABLE table=test-table [id=1e41512af5924c34a8c5b590231793d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:30.447824 7666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab08c1f7dda4ff8aa7968a5c93ee124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:30.447986 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.443673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39715 (local address 127.0.0.1:40150)
0927 11:18:30.443742 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.443752 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:30.443790 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:30.444172 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:30.444173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:30.444174 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:30.444226 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.444227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.445136 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.445137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.446317 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.446319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.447726 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.447727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.447805 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.447808 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:30.447883 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 11:18:30.447926 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":25}
I20260927 11:18:30.448000 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.443947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.0.98.2:39715)
0927 11:18:30.444044 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:30.444047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:30.444055 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:30.444081 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:30.444082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:30.444093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:30.444163 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:30.444276 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.445076 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.447102 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.447647 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.447722 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.447735 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:30.447875 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 11:18:30.447914 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":53,"threads_started":1}
I20260927 11:18:30.449012 7824 tablet_bootstrap.cc:492] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb: Bootstrap starting.
I20260927 11:18:30.449256 7531 tablet_service.cc:1467] Processing CreateTablet for tablet bab08c1f7dda4ff8aa7968a5c93ee124 (DEFAULT_TABLE table=test-table [id=1e41512af5924c34a8c5b590231793d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:30.449398 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab08c1f7dda4ff8aa7968a5c93ee124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:30.449478 7824 tablet_bootstrap.cc:654] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:30.450063 7824 tablet_bootstrap.cc:492] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb: No bootstrap required, opened a new log
I20260927 11:18:30.450126 7824 ts_tablet_manager.cc:1397] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:30.450273 7824 raft_consensus.cc:348] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } }
I20260927 11:18:30.450354 7824 raft_consensus.cc:374] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:30.450381 7824 raft_consensus.cc:729] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a8c2a0a48624715b798b132579a3ceb, State: Initialized, Role: FOLLOWER
I20260927 11:18:30.450424 7824 consensus_queue.cc:260] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } }
I20260927 11:18:30.450577 7824 ts_tablet_manager.cc:1428] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:30.450596 7827 tablet_bootstrap.cc:492] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f: Bootstrap starting.
I20260927 11:18:30.450909 7394 tablet_service.cc:1467] Processing CreateTablet for tablet bab08c1f7dda4ff8aa7968a5c93ee124 (DEFAULT_TABLE table=test-table [id=1e41512af5924c34a8c5b590231793d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:30.451031 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab08c1f7dda4ff8aa7968a5c93ee124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:30.451400 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.451004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.3:56925)
0927 11:18:30.451298 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.451336 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":74,"threads_started":1}
I20260927 11:18:30.451545 7827 tablet_bootstrap.cc:654] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:30.452070 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.451988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.3:35953)
0927 11:18:30.452020 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.452046 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 11:18:30.452164 7827 tablet_bootstrap.cc:492] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f: No bootstrap required, opened a new log
I20260927 11:18:30.452268 7827 ts_tablet_manager.cc:1397] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:30.452353 7735 heartbeater.cc:503] Master 127.0.98.61:33667 was elected leader, sending a full tablet report...
I20260927 11:18:30.452425 7827 raft_consensus.cc:348] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } }
I20260927 11:18:30.452508 7827 raft_consensus.cc:374] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:30.452538 7827 raft_consensus.cc:729] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9bc68d3c124f1da0f058f791c3a21f, State: Initialized, Role: FOLLOWER
I20260927 11:18:30.452577 7827 consensus_queue.cc:260] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } }
I20260927 11:18:30.452689 7830 tablet_bootstrap.cc:492] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606: Bootstrap starting.
I20260927 11:18:30.452735 7827 ts_tablet_manager.cc:1428] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:30.453205 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.452934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.2:48183)
0927 11:18:30.453119 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.453153 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260927 11:18:30.453249 7599 heartbeater.cc:503] Master 127.0.98.61:33667 was elected leader, sending a full tablet report...
I20260927 11:18:30.453408 7830 tablet_bootstrap.cc:654] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:30.453531 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.453442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.2:54047)
0927 11:18:30.453478 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.453505 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 11:18:30.454090 7830 tablet_bootstrap.cc:492] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606: No bootstrap required, opened a new log
I20260927 11:18:30.454133 7830 ts_tablet_manager.cc:1397] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:30.454223 7830 raft_consensus.cc:348] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } }
I20260927 11:18:30.454272 7830 raft_consensus.cc:374] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:30.454284 7830 raft_consensus.cc:729] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd95fd6097a444ac9235305e5ab60606, State: Initialized, Role: FOLLOWER
I20260927 11:18:30.454344 7830 consensus_queue.cc:260] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } }
I20260927 11:18:30.454519 7830 ts_tablet_manager.cc:1428] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260927 11:18:30.454681 7736 tablet.cc:2369] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:30.454941 7462 heartbeater.cc:503] Master 127.0.98.61:33667 was elected leader, sending a full tablet report...
I20260927 11:18:30.455188 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.454975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.1:57291)
0927 11:18:30.455110 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.455145 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260927 11:18:30.455506 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.455422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.1:54529)
0927 11:18:30.455456 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.455478 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260927 11:18:30.648124 7834 raft_consensus.cc:482] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:30.648236 7834 raft_consensus.cc:504] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } }
I20260927 11:18:30.648615 7834 leader_election.cc:288] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9bc68d3c124f1da0f058f791c3a21f (127.0.98.2:39715), 4a8c2a0a48624715b798b132579a3ceb (127.0.98.3:40497)
I20260927 11:18:30.651659 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.648887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39911 (local address 127.0.98.3:40497)
0927 11:18:30.648930 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:30.648932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:30.649087 (+ 155us) server_negotiation.cc:407] Connection header received
0927 11:18:30.649169 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:30.649170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:30.649182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:30.649238 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:30.649388 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.649871 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.650943 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.651291 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.651388 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.651400 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:30.651553 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 11:18:30.651587 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:18:30.651703 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.648880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36039 (local address 127.0.98.2:39715)
0927 11:18:30.648932 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:30.648934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:30.649072 (+ 138us) server_negotiation.cc:407] Connection header received
0927 11:18:30.649158 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:30.649159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:30.649172 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:30.649219 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:30.649406 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.649876 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.650931 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.651319 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.651403 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.651415 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:30.651597 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 11:18:30.651632 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:18:30.651705 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.648802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40497 (local address 127.0.98.1:39911)
0927 11:18:30.648989 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.649006 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:30.649113 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:30.649242 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:30.649243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:30.649244 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:30.649307 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.649309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.649970 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.649971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.650851 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.650853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.651390 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.651391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.651463 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.651467 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:30.651545 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 11:18:30.651584 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":84,"thread_start_us":114,"threads_started":1}
I20260927 11:18:30.652098 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab08c1f7dda4ff8aa7968a5c93ee124" candidate_uuid: "fd95fd6097a444ac9235305e5ab60606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a8c2a0a48624715b798b132579a3ceb" is_pre_election: true
I20260927 11:18:30.652197 7687 raft_consensus.cc:2454] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95fd6097a444ac9235305e5ab60606 in term 0.
I20260927 11:18:30.651708 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.648802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39715 (local address 127.0.98.1:36039)
0927 11:18:30.648975 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.648985 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:30.649092 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:30.649222 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:30.649223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:30.649225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:30.649303 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.649305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.649967 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.649968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.650834 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:30.650837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.651424 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.651426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:30.651509 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.651512 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:30.651586 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 11:18:30.651625 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":33}
I20260927 11:18:30.652410 7350 leader_election.cc:302] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a8c2a0a48624715b798b132579a3ceb, fd95fd6097a444ac9235305e5ab60606; no voters:
I20260927 11:18:30.652403 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab08c1f7dda4ff8aa7968a5c93ee124" candidate_uuid: "fd95fd6097a444ac9235305e5ab60606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" is_pre_election: true
I20260927 11:18:30.652527 7834 raft_consensus.cc:2781] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:30.652572 7834 raft_consensus.cc:482] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:30.652562 7551 raft_consensus.cc:2454] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd95fd6097a444ac9235305e5ab60606 in term 0.
I20260927 11:18:30.652601 7834 raft_consensus.cc:3037] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:30.653363 7834 raft_consensus.cc:504] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } }
I20260927 11:18:30.653517 7834 leader_election.cc:288] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [CANDIDATE]: Term 1 election: Requested vote from peers bf9bc68d3c124f1da0f058f791c3a21f (127.0.98.2:39715), 4a8c2a0a48624715b798b132579a3ceb (127.0.98.3:40497)
I20260927 11:18:30.653681 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab08c1f7dda4ff8aa7968a5c93ee124" candidate_uuid: "fd95fd6097a444ac9235305e5ab60606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9bc68d3c124f1da0f058f791c3a21f"
I20260927 11:18:30.653697 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab08c1f7dda4ff8aa7968a5c93ee124" candidate_uuid: "fd95fd6097a444ac9235305e5ab60606" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a8c2a0a48624715b798b132579a3ceb"
I20260927 11:18:30.653762 7551 raft_consensus.cc:3037] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:30.653761 7687 raft_consensus.cc:3037] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:30.654526 7687 raft_consensus.cc:2454] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95fd6097a444ac9235305e5ab60606 in term 1.
I20260927 11:18:30.654553 7551 raft_consensus.cc:2454] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd95fd6097a444ac9235305e5ab60606 in term 1.
I20260927 11:18:30.654685 7350 leader_election.cc:302] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a8c2a0a48624715b798b132579a3ceb, fd95fd6097a444ac9235305e5ab60606; no voters:
I20260927 11:18:30.654789 7834 raft_consensus.cc:2781] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:30.654937 7834 raft_consensus.cc:686] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [term 1 LEADER]: Becoming Leader. State: Replica: fd95fd6097a444ac9235305e5ab60606, State: Running, Role: LEADER
I20260927 11:18:30.655012 7834 consensus_queue.cc:237] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } }
I20260927 11:18:30.655578 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.655483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.1:52365)
0927 11:18:30.655515 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.655550 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 11:18:30.655748 7093 catalog_manager.cc:5495] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd95fd6097a444ac9235305e5ab60606 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fd95fd6097a444ac9235305e5ab60606" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd95fd6097a444ac9235305e5ab60606" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33883 } health_report { overall_health: HEALTHY } } }
W20260927 11:18:30.662901 7463 tablet.cc:2369] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:30.674273 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.671744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.0.98.1:33883)
0927 11:18:30.671801 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:30.671803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:30.671812 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:30.671885 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:30.671886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:30.671908 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:30.672008 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:30.672149 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.672637 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.673471 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:30.673682 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:30.673767 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:30.673934 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:30.674040 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:30.674115 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:30.674173 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 11:18:30.674193 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:18:30.675292 7687 raft_consensus.cc:1261] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb [term 1 FOLLOWER]: Refusing update from remote peer fd95fd6097a444ac9235305e5ab60606: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:30.675372 7551 raft_consensus.cc:1261] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f [term 1 FOLLOWER]: Refusing update from remote peer fd95fd6097a444ac9235305e5ab60606: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:30.675607 7837 consensus_queue.cc:1035] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a8c2a0a48624715b798b132579a3ceb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:30.675736 7834 consensus_queue.cc:1035] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9bc68d3c124f1da0f058f791c3a21f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:30.676673 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.676558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.3:45405)
0927 11:18:30.676604 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.676637 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 11:18:30.677121 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.676766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.2:54877)
0927 11:18:30.677032 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.677086 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246}
I20260927 11:18:30.677392 7841 mvcc.cc:205] Tried to move back new op lower bound from 7333920402123984896 to 7333920402043490304. Current Snapshot: MvccSnapshot[applied={T|T < 7333920402123984896}]
I20260927 11:18:30.677757 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.677672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.3:36669)
0927 11:18:30.677707 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.677731 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 11:18:30.678251 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.678134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.1:34985)
0927 11:18:30.678193 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.678219 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 11:18:30.678740 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.678631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.2:39521)
0927 11:18:30.678686 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.678716 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 11:18:30.678877 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:30.678634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42973 (local address 127.0.98.1:39757)
0927 11:18:30.678823 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.678857 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260927 11:18:30.680694 7093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:30.680796 7093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:30.682220 7093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6fdfc1f39714f61925ab759c7bfe515: Sending DeleteTablet for 3 replicas of tablet bab08c1f7dda4ff8aa7968a5c93ee124
I20260927 11:18:30.682542 7666 tablet_service.cc:1514] Processing DeleteTablet for tablet bab08c1f7dda4ff8aa7968a5c93ee124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:30 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40790
I20260927 11:18:30.682565 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet bab08c1f7dda4ff8aa7968a5c93ee124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:30 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:32870
I20260927 11:18:30.682713 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet bab08c1f7dda4ff8aa7968a5c93ee124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:30 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40150
I20260927 11:18:30.682790 7850 tablet_replica.cc:330] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb: stopping tablet replica
I20260927 11:18:30.682864 7850 raft_consensus.cc:2229] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:30.682916 7850 raft_consensus.cc:2258] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:30.683015 7849 tablet_replica.cc:330] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606: stopping tablet replica
I20260927 11:18:30.683073 7849 raft_consensus.cc:2229] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:30.683115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 7241
I20260927 11:18:30.683157 7849 raft_consensus.cc:2258] T bab08c1f7dda4ff8aa7968a5c93ee124 P fd95fd6097a444ac9235305e5ab60606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:30.683962 7850 ts_tablet_manager.cc:1905] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:30.685329 7851 tablet_replica.cc:330] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f: stopping tablet replica
I20260927 11:18:30.685398 7851 raft_consensus.cc:2229] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:30.685449 7851 raft_consensus.cc:2258] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:30.685720 7851 ts_tablet_manager.cc:1905] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:30.685951 7850 ts_tablet_manager.cc:1918] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:30.686005 7850 log.cc:1196] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/bab08c1f7dda4ff8aa7968a5c93ee124
I20260927 11:18:30.686210 7850 ts_tablet_manager.cc:1939] T bab08c1f7dda4ff8aa7968a5c93ee124 P 4a8c2a0a48624715b798b132579a3ceb: Deleting consensus metadata
I20260927 11:18:30.686529 7081 catalog_manager.cc:4841] TS 4a8c2a0a48624715b798b132579a3ceb (127.0.98.3:40497): tablet bab08c1f7dda4ff8aa7968a5c93ee124 (table test-table [id=1e41512af5924c34a8c5b590231793d6]) successfully deleted
I20260927 11:18:30.687350 7851 ts_tablet_manager.cc:1918] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:30.687397 7851 log.cc:1196] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/bab08c1f7dda4ff8aa7968a5c93ee124
I20260927 11:18:30.687633 7851 ts_tablet_manager.cc:1939] T bab08c1f7dda4ff8aa7968a5c93ee124 P bf9bc68d3c124f1da0f058f791c3a21f: Deleting consensus metadata
I20260927 11:18:30.687973 7080 catalog_manager.cc:4841] TS bf9bc68d3c124f1da0f058f791c3a21f (127.0.98.2:39715): tablet bab08c1f7dda4ff8aa7968a5c93ee124 (table test-table [id=1e41512af5924c34a8c5b590231793d6]) successfully deleted
W20260927 11:18:30.689414 7804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:30.689205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33883 (local address 127.0.0.1:32890)
0927 11:18:30.689266 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:30.689279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:30.689334 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:30.689374 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33883: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 11:18:30.689471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 7468
W20260927 11:18:30.689599 7081 catalog_manager.cc:4584] TS fd95fd6097a444ac9235305e5ab60606 (127.0.98.1:33883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bab08c1f7dda4ff8aa7968a5c93ee124: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33883: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260927 11:18:30.697039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 7604
I20260927 11:18:30.701697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 6959
I20260927 11:18:30.706461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 7148
2026-09-27T11:18:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5785 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:30 dist-test-slave-lsvg krb5kdc[7854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:30 dist-test-slave-lsvg krb5kdc[7854](info): set up 2 sockets
Sep 27 11:18:30 dist-test-slave-lsvg krb5kdc[7854](info): commencing operation
krb5kdc: starting...
W20260927 11:18:31.745800 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:31 dist-test-slave-lsvg krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 11:18:31.865993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42561
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:31.947209 7867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:31.947410 7867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:31.947432 7867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:31.948621 7867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:31.948668 7867 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 11:18:31.948683 7867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:31.948725 7867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:31.948757 7867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:31.948773 7867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:31.950225 7867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42561
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:31.950464 7867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:31.950706 7867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:31.955624 7875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:31.955638 7873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:31.955639 7872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:31.955878 7867 server_base.cc:1034] running on GCE node
I20260927 11:18:31.956130 7867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:31.956410 7867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:31.957547 7867 hybrid_clock.cc:648] HybridClock initialized: now 1790507911957522 us; error 41 us; skew 500 ppm
Sep 27 11:18:31 dist-test-slave-lsvg krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507911, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:31.959967 7867 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 11:18:31.960436 7867 webserver.cc:460] Webserver started at http://127.0.98.62:37693/ using document root <none> and password file <none>
I20260927 11:18:31.960672 7867 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:31.960714 7867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:31.960861 7867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:31.961948 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "25acfc3c658b4b33894526e0dfda20bf"
format_stamp: "Formatted at 2026-09-27 11:18:31 on dist-test-slave-lsvg"
server_key: "3e036c6e486b5c66fb848f313b8d5381"
server_key_iv: "cbb8ef46d1867871ac6f2c8744e68c41"
server_key_version: "encryptionkey@0"
I20260927 11:18:31.962307 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "25acfc3c658b4b33894526e0dfda20bf"
format_stamp: "Formatted at 2026-09-27 11:18:31 on dist-test-slave-lsvg"
server_key: "3e036c6e486b5c66fb848f313b8d5381"
server_key_iv: "cbb8ef46d1867871ac6f2c8744e68c41"
server_key_version: "encryptionkey@0"
I20260927 11:18:31.963589 7867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:18:31.964267 7882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:31.964434 7867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:31.964533 7867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "25acfc3c658b4b33894526e0dfda20bf"
format_stamp: "Formatted at 2026-09-27 11:18:31 on dist-test-slave-lsvg"
server_key: "3e036c6e486b5c66fb848f313b8d5381"
server_key_iv: "cbb8ef46d1867871ac6f2c8744e68c41"
server_key_version: "encryptionkey@0"
I20260927 11:18:31.964617 7867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:31.981840 7867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:31.983652 7867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:31.983821 7867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:31.988466 7867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42561
I20260927 11:18:31.988511 7944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42561 every 8 connection(s)
I20260927 11:18:31.988831 7867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:31.989465 7945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:31.990968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 7867
I20260927 11:18:31.991080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:31.991282 392 external_mini_cluster.cc:1428] Setting key 142946446241764cd1aea51b11a779ab
I20260927 11:18:31.991822 7945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf: Bootstrap starting.
I20260927 11:18:31.992520 7945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:31.992831 7945 log.cc:824] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:31.993548 7945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf: No bootstrap required, opened a new log
I20260927 11:18:31.995332 7945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25acfc3c658b4b33894526e0dfda20bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42561 } }
I20260927 11:18:31.995488 7945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:31.995535 7945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25acfc3c658b4b33894526e0dfda20bf, State: Initialized, Role: FOLLOWER
I20260927 11:18:31.995677 7945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25acfc3c658b4b33894526e0dfda20bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42561 } }
I20260927 11:18:31.995765 7945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:31.995801 7945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:31.995852 7945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:31.996505 7945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25acfc3c658b4b33894526e0dfda20bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42561 } }
I20260927 11:18:31.996615 7945 leader_election.cc:302] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25acfc3c658b4b33894526e0dfda20bf; no voters:
I20260927 11:18:31.996768 7945 leader_election.cc:288] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:31.996827 7950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:31.996989 7950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [term 1 LEADER]: Becoming Leader. State: Replica: 25acfc3c658b4b33894526e0dfda20bf, State: Running, Role: LEADER
I20260927 11:18:31.997028 7945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:31.997088 7950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25acfc3c658b4b33894526e0dfda20bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42561 } }
I20260927 11:18:31.997409 7951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25acfc3c658b4b33894526e0dfda20bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25acfc3c658b4b33894526e0dfda20bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42561 } } }
I20260927 11:18:31.997483 7951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:31.997701 7952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25acfc3c658b4b33894526e0dfda20bf. Latest consensus state: current_term: 1 leader_uuid: "25acfc3c658b4b33894526e0dfda20bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25acfc3c658b4b33894526e0dfda20bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42561 } } }
I20260927 11:18:31.998021 7952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf [sys.catalog]: This master's current role is: LEADER
Sep 27 11:18:31 dist-test-slave-lsvg krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:18:31.998252 7956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:31.998747 7956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:32.000478 7956 catalog_manager.cc:1347] Generated new cluster ID: d1bd1458207f489f9e044c882d5ae395
I20260927 11:18:32.000541 7956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:32.002542 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:31.991797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.0.98.62:42561)
0927 11:18:31.992010 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:31.992019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:31.992048 (+ 29us) server_negotiation.cc:407] Connection header received
0927 11:18:31.992662 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:31.992672 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:31.992738 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:31.992908 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:31.993699 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:31.994566 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:31.995314 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:31.995770 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:31.995901 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:31.998742 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:31.998746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:31.998746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:31.998758 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:32.000799 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.001186 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.001189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.001190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.001285 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.001453 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.001453 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.001454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.001714 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:32.001826 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.002208 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.002336 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
I20260927 11:18:32.011910 7956 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:32.012426 7956 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:32.018890 7956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf: Generated new TSK 0
I20260927 11:18:32.019073 7956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 11:18:32.047931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42561
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:32.128925 7971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:32.129146 7971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:32.129172 7971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:32.130349 7971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:32.130391 7971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:32.130411 7971 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:18:32.130470 7971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:32.131945 7971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:32.132192 7971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:32.132440 7971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:32.137385 7976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:32.137442 7977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:32.137351 7979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:32.137533 7971 server_base.cc:1034] running on GCE node
I20260927 11:18:32.137758 7971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:32.137974 7971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:32.139110 7971 hybrid_clock.cc:648] HybridClock initialized: now 1790507912139094 us; error 32 us; skew 500 ppm
Sep 27 11:18:32 dist-test-slave-lsvg krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507912, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:32.141649 7971 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 11:18:32.142204 7971 webserver.cc:460] Webserver started at http://127.0.98.1:35409/ using document root <none> and password file <none>
I20260927 11:18:32.142495 7971 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:32.142561 7971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:32.142720 7971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:32.143857 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "1fce593c186f40e88d46e0974b886681"
format_stamp: "Formatted at 2026-09-27 11:18:32 on dist-test-slave-lsvg"
server_key: "922c2d9934304b022e927306bcd27292"
server_key_iv: "60493bb847bfea0b5c32d2ca747e7144"
server_key_version: "encryptionkey@0"
I20260927 11:18:32.144150 7971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1fce593c186f40e88d46e0974b886681"
format_stamp: "Formatted at 2026-09-27 11:18:32 on dist-test-slave-lsvg"
server_key: "922c2d9934304b022e927306bcd27292"
server_key_iv: "60493bb847bfea0b5c32d2ca747e7144"
server_key_version: "encryptionkey@0"
I20260927 11:18:32.145430 7971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:32.146082 7986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.146266 7971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:32.146351 7971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "1fce593c186f40e88d46e0974b886681"
format_stamp: "Formatted at 2026-09-27 11:18:32 on dist-test-slave-lsvg"
server_key: "922c2d9934304b022e927306bcd27292"
server_key_iv: "60493bb847bfea0b5c32d2ca747e7144"
server_key_version: "encryptionkey@0"
I20260927 11:18:32.146418 7971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:32.163535 7971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:32.165951 7971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:32.166108 7971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:32.166608 7971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:32.166648 7971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.166673 7971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:32.166720 7971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.173143 7971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44187
I20260927 11:18:32.173233 8101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44187 every 8 connection(s)
I20260927 11:18:32.173527 7971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:18:32 dist-test-slave-lsvg krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507912, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:18:32.176625 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.167119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58957 (local address 127.0.98.62:42561)
0927 11:18:32.167157 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.167159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.167484 (+ 325us) server_negotiation.cc:407] Connection header received
0927 11:18:32.168038 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.168040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.168052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.168130 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:32.169030 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.169877 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.171422 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.171639 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.171728 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.174354 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:32.174355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:32.174355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:32.174362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:32.175527 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.175880 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.175880 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.175881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.175930 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.176097 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.176097 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.176098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.176188 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:32.176232 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.176482 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.176558 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:18:32.176750 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.167140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42561 (local address 127.0.98.1:58957)
0927 11:18:32.167353 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:32.167395 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:32.167944 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:32.168223 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:32.168228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:32.168593 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:32.168916 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.168923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.169971 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.169972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.171336 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.171339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.171734 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.171735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.171895 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.172468 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:32.172479 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:32.174269 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:32.175599 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.175604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.175605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.175814 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.175971 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.175971 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.175971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.176061 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.176241 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:32.176245 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:32.176368 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 11:18:32.176451 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 11:18:32.176565 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20260927 11:18:32.177265 8102 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42561
I20260927 11:18:32.177356 8102 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:32.177542 8102 heartbeater.cc:511] Master 127.0.98.62:42561 requested a full tablet report, sending...
I20260927 11:18:32.178104 7899 ts_manager.cc:194] Registered new tserver with Master: 1fce593c186f40e88d46e0974b886681 (127.0.98.1:44187)
I20260927 11:18:32.179039 7899 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58957
I20260927 11:18:32.183063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 7971
I20260927 11:18:32.183182 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:32.183403 392 external_mini_cluster.cc:1428] Setting key b80607b31e1a612804b8592c96f858b8
I20260927 11:18:32.183534 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.178377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50939 (local address 127.0.98.62:42561)
0927 11:18:32.178429 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.178431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.178440 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:32.178505 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.178505 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.178512 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.178558 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:32.178871 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.179471 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.180208 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.180391 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.180445 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.181308 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:32.181309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:32.181310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:32.181315 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:32.182416 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.182720 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.182721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.182721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.182786 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.182950 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.182951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.182951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.183097 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:32.183173 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.183359 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.183421 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 11:18:32.183537 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.178250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42561 (local address 127.0.98.1:50939)
0927 11:18:32.178385 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:32.178396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:32.178447 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:32.178564 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:32.178564 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:32.178747 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:32.178811 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.178812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.179562 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.179563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.180134 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.180136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.180457 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.180458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.180503 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.180882 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:32.180886 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:32.181224 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:32.182481 (+ 1257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.182482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.182482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.182635 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.182859 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.182860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.182860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.182909 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.183206 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:32.183208 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:32.183259 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 11:18:32.183334 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 11:18:32.183439 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":27,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 11:18:32.208001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42561
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:32.288528 8111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:32.288769 8111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:32.288810 8111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:32.289986 8111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:32.290056 8111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:32.290091 8111 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:18:32.290127 8111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:32.291613 8111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:32.291913 8111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:32.292145 8111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:32.297127 8116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:32.297127 8119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:32.297151 8117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:32.297253 8111 server_base.cc:1034] running on GCE node
I20260927 11:18:32.297509 8111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:32.297765 8111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:32.298939 8111 hybrid_clock.cc:648] HybridClock initialized: now 1790507912298907 us; error 48 us; skew 500 ppm
Sep 27 11:18:32 dist-test-slave-lsvg krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507912, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:32.301462 8111 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 11:18:32.301905 8111 webserver.cc:460] Webserver started at http://127.0.98.2:36591/ using document root <none> and password file <none>
I20260927 11:18:32.302126 8111 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:32.302170 8111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:32.302322 8111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:32.303344 8111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae8d04c078e3427fbb2eec90a48e0d3a"
format_stamp: "Formatted at 2026-09-27 11:18:32 on dist-test-slave-lsvg"
server_key: "c082683ebbbddccc7edace698e615a35"
server_key_iv: "1249636155bf49d874e97ac6acea2d30"
server_key_version: "encryptionkey@0"
I20260927 11:18:32.303696 8111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae8d04c078e3427fbb2eec90a48e0d3a"
format_stamp: "Formatted at 2026-09-27 11:18:32 on dist-test-slave-lsvg"
server_key: "c082683ebbbddccc7edace698e615a35"
server_key_iv: "1249636155bf49d874e97ac6acea2d30"
server_key_version: "encryptionkey@0"
I20260927 11:18:32.304970 8111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:32.305684 8126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.305853 8111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.305923 8111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "ae8d04c078e3427fbb2eec90a48e0d3a"
format_stamp: "Formatted at 2026-09-27 11:18:32 on dist-test-slave-lsvg"
server_key: "c082683ebbbddccc7edace698e615a35"
server_key_iv: "1249636155bf49d874e97ac6acea2d30"
server_key_version: "encryptionkey@0"
I20260927 11:18:32.306012 8111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:32.318861 8111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:32.321219 8111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:32.321376 8111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:32.321914 8111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:32.321959 8111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.321980 8111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:32.322021 8111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 11:18:32 dist-test-slave-lsvg krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507912, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:18:32.328939 8111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43437
I20260927 11:18:32.329036 8241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43437 every 8 connection(s)
I20260927 11:18:32.329339 8111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:18:32.331686 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.322290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36017 (local address 127.0.98.62:42561)
0927 11:18:32.322346 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.322347 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.322558 (+ 211us) server_negotiation.cc:407] Connection header received
0927 11:18:32.323122 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.323123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.323135 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.323199 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:32.323912 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.324710 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.326425 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.326805 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.326897 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.329301 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:32.329303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:32.329303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:32.329312 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:32.330605 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.330963 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.330964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.330965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.331013 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.331183 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.331184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.331184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.331286 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:32.331342 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.331560 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.331622 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:18:32.331894 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.322298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42561 (local address 127.0.98.2:36017)
0927 11:18:32.322442 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:32.322474 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:32.323015 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:32.323281 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:32.323284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:32.323618 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:32.323817 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.323823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.324829 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.324831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.326317 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.326319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.326906 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.326907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.327035 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.327579 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:32.327637 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:32.329187 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:32.330675 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.330680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.330681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.330896 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.331054 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.331054 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.331054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.331135 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.331344 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:32.331348 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:32.331454 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 11:18:32.331533 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 11:18:32.331644 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260927 11:18:32.332393 8242 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42561
I20260927 11:18:32.332489 8242 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:32.332659 8242 heartbeater.cc:511] Master 127.0.98.62:42561 requested a full tablet report, sending...
I20260927 11:18:32.332913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 8111
I20260927 11:18:32.333067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:32.333040 7898 ts_manager.cc:194] Registered new tserver with Master: ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437)
I20260927 11:18:32.333277 392 external_mini_cluster.cc:1428] Setting key eaa842149197f6e654f0e443a44b701f
I20260927 11:18:32.333542 7898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36017
I20260927 11:18:32.339705 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.333471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42561 (local address 127.0.98.2:45697)
0927 11:18:32.333668 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:32.333678 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:32.333770 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:32.333944 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:32.333945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:32.334155 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:32.334234 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.334236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.334903 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.334904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.335606 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.335608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.336095 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.336096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.336139 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.336582 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:32.336586 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:32.336839 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:32.338745 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.338746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.338746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.338902 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.339105 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.339106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.339106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.339152 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.339483 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:32.339484 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:32.339516 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 11:18:32.339564 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 11:18:32.339633 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":63,"threads_started":1}
I20260927 11:18:32.339782 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.333598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45697 (local address 127.0.98.62:42561)
0927 11:18:32.333632 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.333633 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.333729 (+ 96us) server_negotiation.cc:407] Connection header received
0927 11:18:32.333842 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.333843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.333852 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.333881 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:32.334348 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.334803 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.335687 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.336019 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.336094 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.336922 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:32.336924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:32.336924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:32.336932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:32.338647 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.338974 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.338975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.338976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.339037 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.339212 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.339213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.339213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.339413 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:32.339492 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.339590 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.339668 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 11:18:32.359611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42561
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:32.440506 8251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:32.440701 8251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:32.440724 8251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:32.441922 8251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:32.441964 8251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:32.441984 8251 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:18:32.442028 8251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:32.443470 8251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:32.443707 8251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:32.443938 8251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:32.448901 8259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:32.448891 8256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:32.448982 8251 server_base.cc:1034] running on GCE node
W20260927 11:18:32.448891 8257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:32.449339 8251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:32.449549 8251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:32.450704 8251 hybrid_clock.cc:648] HybridClock initialized: now 1790507912450662 us; error 58 us; skew 500 ppm
Sep 27 11:18:32 dist-test-slave-lsvg krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507912, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:32.453236 8251 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 11:18:32.453707 8251 webserver.cc:460] Webserver started at http://127.0.98.3:37209/ using document root <none> and password file <none>
I20260927 11:18:32.453974 8251 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:32.454041 8251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:32.454191 8251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:32.455227 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "15e42b6a0dbd4000b0f24da694f2b965"
format_stamp: "Formatted at 2026-09-27 11:18:32 on dist-test-slave-lsvg"
server_key: "16b21ea572438dbee549f0d635bfd57c"
server_key_iv: "2d0afb6271ebf41c9d60ba4170d6d1c7"
server_key_version: "encryptionkey@0"
I20260927 11:18:32.455588 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "15e42b6a0dbd4000b0f24da694f2b965"
format_stamp: "Formatted at 2026-09-27 11:18:32 on dist-test-slave-lsvg"
server_key: "16b21ea572438dbee549f0d635bfd57c"
server_key_iv: "2d0afb6271ebf41c9d60ba4170d6d1c7"
server_key_version: "encryptionkey@0"
I20260927 11:18:32.456851 8251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:18:32.457643 8266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.457825 8251 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.457921 8251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "15e42b6a0dbd4000b0f24da694f2b965"
format_stamp: "Formatted at 2026-09-27 11:18:32 on dist-test-slave-lsvg"
server_key: "16b21ea572438dbee549f0d635bfd57c"
server_key_iv: "2d0afb6271ebf41c9d60ba4170d6d1c7"
server_key_version: "encryptionkey@0"
I20260927 11:18:32.458006 8251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:32.470873 8251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:32.473233 8251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:32.473409 8251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:32.473912 8251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:32.473987 8251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.474022 8251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:32.474048 8251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.481041 8251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42627
I20260927 11:18:32.481125 8381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42627 every 8 connection(s)
Sep 27 11:18:32 dist-test-slave-lsvg krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507912, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:18:32.481442 8251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:32.484036 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.474399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36231 (local address 127.0.98.62:42561)
0927 11:18:32.474447 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.474449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.474654 (+ 205us) server_negotiation.cc:407] Connection header received
0927 11:18:32.475180 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.475181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.475194 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.475244 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:32.476106 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.476920 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.478655 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.478967 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.479062 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.481720 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:32.481722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:32.481722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:32.481730 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:32.482953 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.483303 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.483304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.483304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.483350 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.483549 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.483550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.483550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.483645 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:32.483693 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.483916 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.483974 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:18:32.484180 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.474404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42561 (local address 127.0.98.3:36231)
0927 11:18:32.474538 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:32.474570 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:32.475069 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:32.475318 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:32.475321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:32.475674 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:32.475962 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.475974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.477038 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.477040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.478563 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.478566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.479071 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.479072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.479213 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.479766 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:32.479777 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:32.481606 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:32.483021 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.483026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.483028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.483238 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.483406 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.483406 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.483407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.483497 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.483698 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:32.483702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:32.483810 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 11:18:32.483888 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 11:18:32.483999 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":76,"threads_started":1}
I20260927 11:18:32.484243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 8251
I20260927 11:18:32.484349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:32.484588 392 external_mini_cluster.cc:1428] Setting key 3c98348f5869a794cf63dafc1f95ff56
I20260927 11:18:32.484671 8382 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42561
I20260927 11:18:32.484745 8382 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:32.484923 8382 heartbeater.cc:511] Master 127.0.98.62:42561 requested a full tablet report, sending...
I20260927 11:18:32.485301 7898 ts_manager.cc:194] Registered new tserver with Master: 15e42b6a0dbd4000b0f24da694f2b965 (127.0.98.3:42627)
I20260927 11:18:32.485777 7898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36231
I20260927 11:18:32.490577 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.485789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44885 (local address 127.0.98.62:42561)
0927 11:18:32.485829 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.485831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.485906 (+ 75us) server_negotiation.cc:407] Connection header received
0927 11:18:32.485985 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.485985 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.485995 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.486052 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:32.486355 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.486888 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.487512 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.487677 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.487722 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.488425 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:32.488426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:32.488426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:32.488431 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:32.489777 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.490064 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.490065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.490065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.490107 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.490248 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.490248 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.490248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.490326 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:32.490369 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.490474 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.490521 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:18:32.490594 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.485696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42561 (local address 127.0.98.3:44885)
0927 11:18:32.485830 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:32.485842 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:32.485905 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:32.486055 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:32.486056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:32.486206 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:32.486285 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.486286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.486975 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.486975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.487449 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.487450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.487735 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.487736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.487785 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.488113 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:32.488116 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:32.488356 (+ 240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:32.489841 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.489841 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.489841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.489989 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.490159 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.490160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.490160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.490207 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.490375 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:32.490375 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:32.490412 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 11:18:32.490459 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 11:18:32.490526 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260927 11:18:32.496184 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 11:18:32 dist-test-slave-lsvg krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507912, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 11:18:32 dist-test-slave-lsvg krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:32 dist-test-slave-lsvg krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 11:18:32.516721 7948 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:32.511410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.0.98.62:42561)
0927 11:18:32.511477 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.511479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.511490 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:18:32.511537 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.511538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.511553 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.511615 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:32.512140 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.512628 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.513466 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.513656 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.513743 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.516644 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38304: BlockingRecv error: recv got EOF from 127.0.0.1:38304 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 11:18:32 dist-test-slave-lsvg krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507912, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:18:32.523340 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.517493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.0.98.62:42561)
0927 11:18:32.517528 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.517530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.517606 (+ 76us) server_negotiation.cc:407] Connection header received
0927 11:18:32.517641 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.517642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.517649 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.517700 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:32.518108 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.518552 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.519187 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.519383 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.519436 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.521396 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:32.521397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:32.521397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:32.521403 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:32.522502 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.522789 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.522790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.522790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.522838 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.522989 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.522989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.522990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.523082 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:32.523154 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.523227 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.523277 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:18:32.524853 7898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:32.526499 7898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:32.532773 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.528973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57768 (local address 127.0.98.1:44187)
0927 11:18:32.529196 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.529200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.529208 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:32.529238 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.529240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.529253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.529346 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:32.529499 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.530608 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.531705 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.532275 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.532340 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.532412 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.532624 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.532663 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":91,"threads_started":1}
I20260927 11:18:32.532789 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.528777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42627 (local address 127.0.0.1:58280)
0927 11:18:32.529166 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:32.529175 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:32.529219 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:32.529481 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:32.529482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:32.529484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:32.529535 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.529537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.530436 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.530437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.531489 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.531491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.532469 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.532470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.532530 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.532540 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 11:18:32.532604 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 11:18:32.532667 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":86,"threads_started":1}
I20260927 11:18:32.533089 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.528987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.0.98.3:42627)
0927 11:18:32.529337 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.529346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.529354 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:32.529384 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.529386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.529398 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.529483 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:32.529614 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.530340 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.531583 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.532392 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.532471 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.532541 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.532923 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.532970 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":102,"threads_started":1}
I20260927 11:18:32.532776 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.528764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44187 (local address 127.0.0.1:57768)
0927 11:18:32.528917 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:32.528955 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:32.529023 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:32.529348 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:32.529350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:32.529353 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:32.529415 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.529417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.530708 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.530709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.531621 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.531624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.532446 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.532447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.532523 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.532536 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 11:18:32.532606 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 11:18:32.532676 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":28,"thread_start_us":53,"threads_started":1}
I20260927 11:18:32.533319 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.528774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43437 (local address 127.0.0.1:55614)
0927 11:18:32.529150 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:32.529161 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:32.529219 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:32.529787 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:32.529788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:32.529790 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:32.529842 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.529843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.531013 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.531014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.532038 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.532040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.533029 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.533030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.533151 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.533155 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:32.533211 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 11:18:32.533249 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":77,"threads_started":1}
I20260927 11:18:32.533412 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.528949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.0.98.2:43437)
0927 11:18:32.529460 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.529579 (+ 119us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.529589 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:32.529629 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.529631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.529646 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.529714 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:32.529912 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.530931 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.532124 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.532954 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.533025 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.533109 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.533228 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.533271 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":125,"threads_started":1}
I20260927 11:18:32.533810 8316 tablet_service.cc:1467] Processing CreateTablet for tablet aef40248d470499ebf5a8075c98c4be9 (DEFAULT_TABLE table=test-table [id=122c69219dc04d1cacd54b5c84ac4a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:32.533807 8036 tablet_service.cc:1467] Processing CreateTablet for tablet aef40248d470499ebf5a8075c98c4be9 (DEFAULT_TABLE table=test-table [id=122c69219dc04d1cacd54b5c84ac4a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:32.533937 8176 tablet_service.cc:1467] Processing CreateTablet for tablet aef40248d470499ebf5a8075c98c4be9 (DEFAULT_TABLE table=test-table [id=122c69219dc04d1cacd54b5c84ac4a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:32.534142 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef40248d470499ebf5a8075c98c4be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:32.534142 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef40248d470499ebf5a8075c98c4be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:32.534170 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef40248d470499ebf5a8075c98c4be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:32.536897 8410 tablet_bootstrap.cc:492] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a: Bootstrap starting.
I20260927 11:18:32.537490 8410 tablet_bootstrap.cc:654] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:32.537776 8410 log.cc:824] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:32.537791 8411 tablet_bootstrap.cc:492] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965: Bootstrap starting.
I20260927 11:18:32.538179 8412 tablet_bootstrap.cc:492] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681: Bootstrap starting.
I20260927 11:18:32.538352 8411 tablet_bootstrap.cc:654] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:32.538669 8411 log.cc:824] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:32.538779 8412 tablet_bootstrap.cc:654] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:32.539018 8410 tablet_bootstrap.cc:492] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a: No bootstrap required, opened a new log
I20260927 11:18:32.539095 8410 ts_tablet_manager.cc:1397] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:32.539091 8412 log.cc:824] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:32.539438 8411 tablet_bootstrap.cc:492] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965: No bootstrap required, opened a new log
I20260927 11:18:32.539515 8411 ts_tablet_manager.cc:1397] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:32.539981 8412 tablet_bootstrap.cc:492] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681: No bootstrap required, opened a new log
I20260927 11:18:32.540050 8412 ts_tablet_manager.cc:1397] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:32.540661 8410 raft_consensus.cc:348] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.540843 8410 raft_consensus.cc:374] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:32.540886 8410 raft_consensus.cc:729] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8d04c078e3427fbb2eec90a48e0d3a, State: Initialized, Role: FOLLOWER
I20260927 11:18:32.541006 8410 consensus_queue.cc:260] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.541320 8410 ts_tablet_manager.cc:1428] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:18:32.541437 8242 heartbeater.cc:503] Master 127.0.98.62:42561 was elected leader, sending a full tablet report...
I20260927 11:18:32.541546 8411 raft_consensus.cc:348] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.541693 8411 raft_consensus.cc:374] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:32.541742 8411 raft_consensus.cc:729] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e42b6a0dbd4000b0f24da694f2b965, State: Initialized, Role: FOLLOWER
I20260927 11:18:32.541850 8411 consensus_queue.cc:260] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.541975 8412 raft_consensus.cc:348] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.542090 8411 ts_tablet_manager.cc:1428] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:18:32.542111 8412 raft_consensus.cc:374] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:32.542145 8412 raft_consensus.cc:729] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fce593c186f40e88d46e0974b886681, State: Initialized, Role: FOLLOWER
I20260927 11:18:32.542136 8382 heartbeater.cc:503] Master 127.0.98.62:42561 was elected leader, sending a full tablet report...
I20260927 11:18:32.542266 8412 consensus_queue.cc:260] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.542523 8412 ts_tablet_manager.cc:1428] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:18:32.542572 8102 heartbeater.cc:503] Master 127.0.98.62:42561 was elected leader, sending a full tablet report...
W20260927 11:18:32.579907 8243 tablet.cc:2369] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:32.640609 8417 raft_consensus.cc:482] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:32.640769 8417 raft_consensus.cc:504] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.641197 8417 leader_election.cc:288] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437), 1fce593c186f40e88d46e0974b886681 (127.0.98.1:44187)
I20260927 11:18:32.644173 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.641564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45781 (local address 127.0.98.1:44187)
0927 11:18:32.641622 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.641624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.641636 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:18:32.641707 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.641708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.641724 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.641801 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:32.641964 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.642450 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.643437 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.643816 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.643902 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.643911 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.644068 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.644106 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:18:32.644204 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.641432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44187 (local address 127.0.98.3:45781)
0927 11:18:32.641536 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:32.641550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:32.641622 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:32.641807 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:32.641808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:32.641809 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:32.641888 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.641890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.642537 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.642538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.643340 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.643342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.643902 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.643903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.643986 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.643988 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:18:32.644051 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 11:18:32.644099 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":34}
I20260927 11:18:32.644220 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.641432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43437 (local address 127.0.98.3:58609)
0927 11:18:32.641714 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:32.641722 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:32.641816 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:32.641969 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:32.641970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:32.641972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:32.642055 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.642057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.642679 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.642679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.643418 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.643420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.643957 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.643958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.644026 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.644029 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:32.644090 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 11:18:32.644129 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":98,"thread_start_us":120,"threads_started":1}
I20260927 11:18:32.644209 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.641586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58609 (local address 127.0.98.2:43437)
0927 11:18:32.641633 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.641635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.641806 (+ 171us) server_negotiation.cc:407] Connection header received
0927 11:18:32.641885 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.641886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.641903 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.641967 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:32.642134 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.642591 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.643509 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.643864 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.643942 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.643952 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.644109 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.644141 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 11:18:32.644569 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef40248d470499ebf5a8075c98c4be9" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fce593c186f40e88d46e0974b886681" is_pre_election: true
I20260927 11:18:32.644759 8056 raft_consensus.cc:2454] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 0.
I20260927 11:18:32.644757 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef40248d470499ebf5a8075c98c4be9" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" is_pre_election: true
I20260927 11:18:32.644876 8196 raft_consensus.cc:2454] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 0.
I20260927 11:18:32.645089 8269 leader_election.cc:302] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e42b6a0dbd4000b0f24da694f2b965, 1fce593c186f40e88d46e0974b886681; no voters:
I20260927 11:18:32.645278 8417 raft_consensus.cc:2781] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:32.645344 8417 raft_consensus.cc:482] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:32.645378 8417 raft_consensus.cc:3037] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:32.646327 8417 raft_consensus.cc:504] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.646476 8417 leader_election.cc:288] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 election: Requested vote from peers ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437), 1fce593c186f40e88d46e0974b886681 (127.0.98.1:44187)
I20260927 11:18:32.646625 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef40248d470499ebf5a8075c98c4be9" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a"
I20260927 11:18:32.646636 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef40248d470499ebf5a8075c98c4be9" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fce593c186f40e88d46e0974b886681"
I20260927 11:18:32.646706 8196 raft_consensus.cc:3037] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:32.646726 8056 raft_consensus.cc:3037] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:32.647400 8196 raft_consensus.cc:2454] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 1.
I20260927 11:18:32.647509 8056 raft_consensus.cc:2454] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 1.
I20260927 11:18:32.647656 8267 leader_election.cc:302] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e42b6a0dbd4000b0f24da694f2b965, ae8d04c078e3427fbb2eec90a48e0d3a; no voters:
I20260927 11:18:32.647775 8417 raft_consensus.cc:2781] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:32.647924 8417 raft_consensus.cc:686] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 LEADER]: Becoming Leader. State: Replica: 15e42b6a0dbd4000b0f24da694f2b965, State: Running, Role: LEADER
I20260927 11:18:32.648016 8417 consensus_queue.cc:237] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.648919 7898 catalog_manager.cc:5495] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15e42b6a0dbd4000b0f24da694f2b965 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "15e42b6a0dbd4000b0f24da694f2b965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } health_report { overall_health: HEALTHY } } }
W20260927 11:18:32.674335 8103 tablet.cc:2369] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:32.681680 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.679491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.0.98.3:42627)
0927 11:18:32.679549 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.679551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.679560 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:32.679609 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.679610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.679629 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.679693 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:32.679800 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.680256 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.680948 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.681216 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.681273 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.681398 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:32.681493 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:32.681548 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.681581 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.681624 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:18:32.683444 8056 raft_consensus.cc:1261] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Refusing update from remote peer 15e42b6a0dbd4000b0f24da694f2b965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:32.683444 8196 raft_consensus.cc:1261] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Refusing update from remote peer 15e42b6a0dbd4000b0f24da694f2b965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:32.683799 8417 consensus_queue.cc:1035] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:32.683965 8420 consensus_queue.cc:1035] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:32.687037 8425 mvcc.cc:205] Tried to move back new op lower bound from 7333920410348666880 to 7333920410206961664. Current Snapshot: MvccSnapshot[applied={T|T < 7333920410348666880}]
I20260927 11:18:32.687151 8424 mvcc.cc:205] Tried to move back new op lower bound from 7333920410348666880 to 7333920410206961664. Current Snapshot: MvccSnapshot[applied={T|T < 7333920410348666880 or (T in {7333920410348666880})}]
I20260927 11:18:32.688661 7898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:32.688773 7898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:32.689558 7898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf: Sending DeleteTablet for 3 replicas of tablet aef40248d470499ebf5a8075c98c4be9
I20260927 11:18:32.689910 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet aef40248d470499ebf5a8075c98c4be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57768
I20260927 11:18:32.690002 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet aef40248d470499ebf5a8075c98c4be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55614
I20260927 11:18:32.690120 8433 tablet_replica.cc:330] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681: stopping tablet replica
I20260927 11:18:32.690232 8433 raft_consensus.cc:2229] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:32.690223 8316 tablet_service.cc:1514] Processing DeleteTablet for tablet aef40248d470499ebf5a8075c98c4be9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58280
I20260927 11:18:32.690346 8433 raft_consensus.cc:2258] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:32.690547 8435 tablet_replica.cc:330] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965: stopping tablet replica
I20260927 11:18:32.690688 8435 raft_consensus.cc:2229] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:32.690708 7898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 11:18:32.690804 8433 ts_tablet_manager.cc:1905] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 11:18:32.690855 7898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:32.691023 8435 raft_consensus.cc:2258] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:32.691522 8435 ts_tablet_manager.cc:1905] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:32.692185 8036 tablet_service.cc:1467] Processing CreateTablet for tablet 98fe60ba926b4b219ee36f9d950cf0c7 (DEFAULT_TABLE table=test-table [id=f2d7816ed56f4de6942b7a8cc49c2e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:32.692308 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fe60ba926b4b219ee36f9d950cf0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:32.692652 8316 tablet_service.cc:1467] Processing CreateTablet for tablet 98fe60ba926b4b219ee36f9d950cf0c7 (DEFAULT_TABLE table=test-table [id=f2d7816ed56f4de6942b7a8cc49c2e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:32.692802 8176 tablet_service.cc:1467] Processing CreateTablet for tablet 98fe60ba926b4b219ee36f9d950cf0c7 (DEFAULT_TABLE table=test-table [id=f2d7816ed56f4de6942b7a8cc49c2e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:32.692912 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fe60ba926b4b219ee36f9d950cf0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:32.693698 8412 tablet_bootstrap.cc:492] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681: Bootstrap starting.
I20260927 11:18:32.693809 8433 ts_tablet_manager.cc:1918] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:32.693874 8433 log.cc:1196] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/wal/wals/aef40248d470499ebf5a8075c98c4be9
I20260927 11:18:32.694065 8412 tablet_bootstrap.cc:654] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:32.694131 8433 ts_tablet_manager.cc:1939] T aef40248d470499ebf5a8075c98c4be9 P 1fce593c186f40e88d46e0974b886681: Deleting consensus metadata
I20260927 11:18:32.694415 8410 tablet_bootstrap.cc:492] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a: Bootstrap starting.
I20260927 11:18:32.694499 7885 catalog_manager.cc:4841] TS 1fce593c186f40e88d46e0974b886681 (127.0.98.1:44187): tablet aef40248d470499ebf5a8075c98c4be9 (table test-table [id=122c69219dc04d1cacd54b5c84ac4a8e]) successfully deleted
I20260927 11:18:32.694816 8410 tablet_bootstrap.cc:654] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:32.694967 8412 tablet_bootstrap.cc:492] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681: No bootstrap required, opened a new log
I20260927 11:18:32.695006 8412 ts_tablet_manager.cc:1397] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:32.695254 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fe60ba926b4b219ee36f9d950cf0c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:32.695309 8412 raft_consensus.cc:348] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.695376 8412 raft_consensus.cc:374] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:32.695397 8412 raft_consensus.cc:729] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fce593c186f40e88d46e0974b886681, State: Initialized, Role: FOLLOWER
I20260927 11:18:32.695444 8412 consensus_queue.cc:260] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.695535 8412 ts_tablet_manager.cc:1428] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.696506 8411 tablet_bootstrap.cc:492] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965: Bootstrap starting.
I20260927 11:18:32.696957 8411 tablet_bootstrap.cc:654] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:32.696959 8410 tablet_bootstrap.cc:492] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a: No bootstrap required, opened a new log
I20260927 11:18:32.697010 8410 ts_tablet_manager.cc:1397] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 11:18:32.697199 8410 raft_consensus.cc:348] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.697306 8410 raft_consensus.cc:374] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:32.697330 8410 raft_consensus.cc:729] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8d04c078e3427fbb2eec90a48e0d3a, State: Initialized, Role: FOLLOWER
I20260927 11:18:32.697378 8410 consensus_queue.cc:260] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.697485 8410 ts_tablet_manager.cc:1428] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:32.697804 8411 tablet_bootstrap.cc:492] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965: No bootstrap required, opened a new log
I20260927 11:18:32.697861 8411 ts_tablet_manager.cc:1397] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:32.698007 8411 raft_consensus.cc:348] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.698091 8411 raft_consensus.cc:374] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:32.698117 8411 raft_consensus.cc:729] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e42b6a0dbd4000b0f24da694f2b965, State: Initialized, Role: FOLLOWER
I20260927 11:18:32.698159 8411 consensus_queue.cc:260] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.698282 8411 ts_tablet_manager.cc:1428] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.698437 8435 ts_tablet_manager.cc:1918] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:32.698508 8435 log.cc:1196] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/aef40248d470499ebf5a8075c98c4be9
I20260927 11:18:32.698765 8435 ts_tablet_manager.cc:1939] T aef40248d470499ebf5a8075c98c4be9 P 15e42b6a0dbd4000b0f24da694f2b965: Deleting consensus metadata
I20260927 11:18:32.699273 8434 tablet_replica.cc:330] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a: stopping tablet replica
I20260927 11:18:32.699271 7886 catalog_manager.cc:4841] TS 15e42b6a0dbd4000b0f24da694f2b965 (127.0.98.3:42627): tablet aef40248d470499ebf5a8075c98c4be9 (table test-table [id=122c69219dc04d1cacd54b5c84ac4a8e]) successfully deleted
I20260927 11:18:32.699375 8434 raft_consensus.cc:2229] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:32.699456 8434 raft_consensus.cc:2258] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:32.699812 8434 ts_tablet_manager.cc:1905] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:32.701081 8434 ts_tablet_manager.cc:1918] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:32.701156 8434 log.cc:1196] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/aef40248d470499ebf5a8075c98c4be9
I20260927 11:18:32.701436 8434 ts_tablet_manager.cc:1939] T aef40248d470499ebf5a8075c98c4be9 P ae8d04c078e3427fbb2eec90a48e0d3a: Deleting consensus metadata
I20260927 11:18:32.701757 7883 catalog_manager.cc:4841] TS ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437): tablet aef40248d470499ebf5a8075c98c4be9 (table test-table [id=122c69219dc04d1cacd54b5c84ac4a8e]) successfully deleted
W20260927 11:18:32.732321 8383 tablet.cc:2369] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:32.898948 8428 raft_consensus.cc:482] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:32.899041 8428 raft_consensus.cc:504] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.899228 8428 leader_election.cc:288] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fce593c186f40e88d46e0974b886681 (127.0.98.1:44187), ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437)
I20260927 11:18:32.899502 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe60ba926b4b219ee36f9d950cf0c7" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fce593c186f40e88d46e0974b886681" is_pre_election: true
I20260927 11:18:32.899501 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe60ba926b4b219ee36f9d950cf0c7" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" is_pre_election: true
I20260927 11:18:32.899588 8056 raft_consensus.cc:2454] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 0.
I20260927 11:18:32.899608 8196 raft_consensus.cc:2454] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 0.
I20260927 11:18:32.899772 8267 leader_election.cc:302] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e42b6a0dbd4000b0f24da694f2b965, ae8d04c078e3427fbb2eec90a48e0d3a; no voters:
I20260927 11:18:32.899858 8428 raft_consensus.cc:2781] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:32.899899 8428 raft_consensus.cc:482] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:32.899919 8428 raft_consensus.cc:3037] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:32.900590 8428 raft_consensus.cc:504] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.900748 8428 leader_election.cc:288] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 election: Requested vote from peers 1fce593c186f40e88d46e0974b886681 (127.0.98.1:44187), ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437)
I20260927 11:18:32.900873 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe60ba926b4b219ee36f9d950cf0c7" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fce593c186f40e88d46e0974b886681"
I20260927 11:18:32.900897 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fe60ba926b4b219ee36f9d950cf0c7" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a"
I20260927 11:18:32.900970 8196 raft_consensus.cc:3037] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:32.900974 8056 raft_consensus.cc:3037] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:32.901563 8196 raft_consensus.cc:2454] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 1.
I20260927 11:18:32.901677 8056 raft_consensus.cc:2454] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 1.
I20260927 11:18:32.901723 8267 leader_election.cc:302] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e42b6a0dbd4000b0f24da694f2b965, ae8d04c078e3427fbb2eec90a48e0d3a; no voters:
I20260927 11:18:32.901825 8428 raft_consensus.cc:2781] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:32.901892 8428 raft_consensus.cc:686] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 LEADER]: Becoming Leader. State: Replica: 15e42b6a0dbd4000b0f24da694f2b965, State: Running, Role: LEADER
I20260927 11:18:32.901945 8428 consensus_queue.cc:237] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:32.902544 7898 catalog_manager.cc:5495] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15e42b6a0dbd4000b0f24da694f2b965 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "15e42b6a0dbd4000b0f24da694f2b965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } health_report { overall_health: HEALTHY } } }
Sep 27 11:18:32 dist-test-slave-lsvg krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507911, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:18:32.953897 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.941795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.0.98.62:42561)
0927 11:18:32.941857 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:32.941859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:32.941988 (+ 129us) server_negotiation.cc:407] Connection header received
0927 11:18:32.942030 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:32.942031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:32.942040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:32.942091 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:32.942526 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.943004 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.943662 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.943843 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.943894 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.951595 (+ 7701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:32.951596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:32.951597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:32.951603 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:32.952916 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.953295 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.953296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.953296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.953343 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:32.953501 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:32.953501 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:32.953502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:32.953593 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:32.953637 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:32.953770 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 11:18:32.953832 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 11:18:32.954056 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:32.941766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42561 (local address 127.0.0.1:38310)
0927 11:18:32.941913 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:32.941922 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:32.941988 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:32.942108 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:32.942108 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:32.942419 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:32.942465 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.942467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.943069 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.943070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.943600 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:32.943601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:32.943905 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:32.943906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:32.943944 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:32.950187 (+ 6243us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:32.950191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:32.951512 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:32.952984 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.952990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.952992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.953225 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.953401 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:32.953401 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:32.953402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:32.953455 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:32.953641 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:32.953646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:32.953687 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 11:18:32.953752 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 11:18:32.953901 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":62,"thread_start_us":87,"threads_started":1}
I20260927 11:18:32.955808 7898 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38310:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 11:18:32.956004 7898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:32.957443 8176 tablet_service.cc:1467] Processing CreateTablet for tablet b0d12efb51814b5e91a1cc2aaf453015 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=961674ceaf864a97a5e21b87e3ac3d80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:18:32.957468 8316 tablet_service.cc:1467] Processing CreateTablet for tablet b0d12efb51814b5e91a1cc2aaf453015 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=961674ceaf864a97a5e21b87e3ac3d80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:18:32.957474 8036 tablet_service.cc:1467] Processing CreateTablet for tablet b0d12efb51814b5e91a1cc2aaf453015 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=961674ceaf864a97a5e21b87e3ac3d80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:18:32.957608 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d12efb51814b5e91a1cc2aaf453015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:32.957608 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d12efb51814b5e91a1cc2aaf453015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:32.957615 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d12efb51814b5e91a1cc2aaf453015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:32.958737 8412 tablet_bootstrap.cc:492] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681: Bootstrap starting.
I20260927 11:18:32.959038 8410 tablet_bootstrap.cc:492] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a: Bootstrap starting.
I20260927 11:18:32.959043 8411 tablet_bootstrap.cc:492] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965: Bootstrap starting.
I20260927 11:18:32.959151 8412 tablet_bootstrap.cc:654] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:32.959381 8411 tablet_bootstrap.cc:654] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:32.959455 8410 tablet_bootstrap.cc:654] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:32.959847 8412 tablet_bootstrap.cc:492] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681: No bootstrap required, opened a new log
I20260927 11:18:32.959899 8412 ts_tablet_manager.cc:1397] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:32.959972 8411 tablet_bootstrap.cc:492] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965: No bootstrap required, opened a new log
I20260927 11:18:32.960016 8410 tablet_bootstrap.cc:492] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a: No bootstrap required, opened a new log
I20260927 11:18:32.960022 8411 ts_tablet_manager.cc:1397] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:32.960062 8410 ts_tablet_manager.cc:1397] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:32.960062 8412 raft_consensus.cc:348] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } }
I20260927 11:18:32.960144 8412 raft_consensus.cc:374] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:32.960126 8411 raft_consensus.cc:348] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } }
I20260927 11:18:32.960171 8412 raft_consensus.cc:729] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fce593c186f40e88d46e0974b886681, State: Initialized, Role: FOLLOWER
I20260927 11:18:32.960189 8411 raft_consensus.cc:374] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:32.960208 8411 raft_consensus.cc:729] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e42b6a0dbd4000b0f24da694f2b965, State: Initialized, Role: FOLLOWER
I20260927 11:18:32.960210 8412 consensus_queue.cc:260] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } }
I20260927 11:18:32.960206 8410 raft_consensus.cc:348] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } }
I20260927 11:18:32.960242 8411 consensus_queue.cc:260] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } }
I20260927 11:18:32.960275 8410 raft_consensus.cc:374] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:32.960295 8410 raft_consensus.cc:729] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8d04c078e3427fbb2eec90a48e0d3a, State: Initialized, Role: FOLLOWER
I20260927 11:18:32.960301 8412 ts_tablet_manager.cc:1428] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.960325 8411 ts_tablet_manager.cc:1428] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:32.960340 8410 consensus_queue.cc:260] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } }
I20260927 11:18:32.960409 8428 tablet_replica.cc:439] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } }
I20260927 11:18:32.960441 8410 ts_tablet_manager.cc:1428] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:32.960481 8428 tablet_replica.cc:442] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:32.960453 8418 tablet_replica.cc:439] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } }
I20260927 11:18:32.960534 8418 tablet_replica.cc:442] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:32.960680 8416 tablet_replica.cc:439] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } }
I20260927 11:18:32.960777 8416 tablet_replica.cc:442] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:33.130074 8428 raft_consensus.cc:482] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:33.130175 8428 raft_consensus.cc:504] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } }
I20260927 11:18:33.130358 8428 leader_election.cc:288] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fce593c186f40e88d46e0974b886681 (127.0.98.1:44187), ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437)
I20260927 11:18:33.130611 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d12efb51814b5e91a1cc2aaf453015" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fce593c186f40e88d46e0974b886681" is_pre_election: true
I20260927 11:18:33.130627 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d12efb51814b5e91a1cc2aaf453015" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" is_pre_election: true
I20260927 11:18:33.130714 8056 raft_consensus.cc:2454] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 0.
I20260927 11:18:33.130723 8196 raft_consensus.cc:2454] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 0.
I20260927 11:18:33.130888 8267 leader_election.cc:302] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e42b6a0dbd4000b0f24da694f2b965, ae8d04c078e3427fbb2eec90a48e0d3a; no voters:
I20260927 11:18:33.131009 8428 raft_consensus.cc:2781] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:33.131048 8428 raft_consensus.cc:482] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:33.131072 8428 raft_consensus.cc:3037] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:33.131767 8428 raft_consensus.cc:504] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } }
I20260927 11:18:33.131903 8428 leader_election.cc:288] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 election: Requested vote from peers 1fce593c186f40e88d46e0974b886681 (127.0.98.1:44187), ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437)
I20260927 11:18:33.132036 8056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d12efb51814b5e91a1cc2aaf453015" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fce593c186f40e88d46e0974b886681"
I20260927 11:18:33.132045 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d12efb51814b5e91a1cc2aaf453015" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a"
I20260927 11:18:33.132138 8056 raft_consensus.cc:3037] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:33.132169 8196 raft_consensus.cc:3037] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:33.132771 8056 raft_consensus.cc:2454] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 1.
I20260927 11:18:33.132892 8196 raft_consensus.cc:2454] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 1.
I20260927 11:18:33.132975 8269 leader_election.cc:302] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e42b6a0dbd4000b0f24da694f2b965, 1fce593c186f40e88d46e0974b886681; no voters:
I20260927 11:18:33.133082 8428 raft_consensus.cc:2781] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:33.133127 8428 raft_consensus.cc:686] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 LEADER]: Becoming Leader. State: Replica: 15e42b6a0dbd4000b0f24da694f2b965, State: Running, Role: LEADER
I20260927 11:18:33.133177 8428 consensus_queue.cc:237] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } }
I20260927 11:18:33.133450 8417 tablet_replica.cc:439] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965: TxnStatusTablet state changed. Reason: New leader 15e42b6a0dbd4000b0f24da694f2b965. Latest consensus state: current_term: 1 leader_uuid: "15e42b6a0dbd4000b0f24da694f2b965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } }
I20260927 11:18:33.133520 8417 tablet_replica.cc:442] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:18:33.133673 8450 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 11:18:33.133805 8450 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 11:18:33.133775 7898 catalog_manager.cc:5495] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15e42b6a0dbd4000b0f24da694f2b965 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "15e42b6a0dbd4000b0f24da694f2b965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } health_report { overall_health: UNKNOWN } } }
I20260927 11:18:33.142678 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:33.140330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.0.98.3:42627)
0927 11:18:33.140380 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:33.140382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:33.140392 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:33.140465 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:33.140466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:33.140476 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:33.140531 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:33.140663 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.141207 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.141961 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.142163 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.142253 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:33.142419 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:33.142453 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:33.142523 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:33.142586 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 11:18:33.142606 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:18:33.142690 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:33.140266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42627 (local address 127.0.0.1:58292)
0927 11:18:33.140338 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:33.140351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:33.140400 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:33.140538 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:33.140539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:33.140541 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:18:33.140588 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:33.140590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.141287 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.141288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:33.141873 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:33.141875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.142254 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.142255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:33.142320 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:33.142339 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:18:33.142523 (+ 184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:18:33.142531 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 11:18:33.142573 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 11:18:33.142623 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 11:18:33.143692 8056 raft_consensus.cc:1261] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Refusing update from remote peer 15e42b6a0dbd4000b0f24da694f2b965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:33.143779 8196 raft_consensus.cc:1261] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Refusing update from remote peer 15e42b6a0dbd4000b0f24da694f2b965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:33.143919 8417 consensus_queue.cc:1035] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:33.143998 8428 consensus_queue.cc:1035] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:33.144541 8418 tablet_replica.cc:439] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681: TxnStatusTablet state changed. Reason: New leader 15e42b6a0dbd4000b0f24da694f2b965. Latest consensus state: current_term: 1 leader_uuid: "15e42b6a0dbd4000b0f24da694f2b965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } }
I20260927 11:18:33.144629 8418 tablet_replica.cc:442] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:33.144698 8416 tablet_replica.cc:439] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a: TxnStatusTablet state changed. Reason: New leader 15e42b6a0dbd4000b0f24da694f2b965. Latest consensus state: current_term: 1 leader_uuid: "15e42b6a0dbd4000b0f24da694f2b965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } }
I20260927 11:18:33.144786 8416 tablet_replica.cc:442] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:33.145205 8420 tablet_replica.cc:439] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15e42b6a0dbd4000b0f24da694f2b965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } }
I20260927 11:18:33.145298 8420 tablet_replica.cc:442] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:18:33.145346 8423 mvcc.cc:205] Tried to move back new op lower bound from 7333920412234907648 to 7333920412193968128. Current Snapshot: MvccSnapshot[applied={T|T < 7333920412234907648 or (T in {7333920412234907648})}]
I20260927 11:18:33.145677 8416 tablet_replica.cc:439] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15e42b6a0dbd4000b0f24da694f2b965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } }
I20260927 11:18:33.145750 8416 tablet_replica.cc:442] T b0d12efb51814b5e91a1cc2aaf453015 P ae8d04c078e3427fbb2eec90a48e0d3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:33.146608 8428 tablet_replica.cc:439] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15e42b6a0dbd4000b0f24da694f2b965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } }
I20260927 11:18:33.146684 8428 tablet_replica.cc:442] T b0d12efb51814b5e91a1cc2aaf453015 P 15e42b6a0dbd4000b0f24da694f2b965: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:18:33.147071 8418 tablet_replica.cc:439] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15e42b6a0dbd4000b0f24da694f2b965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } }
I20260927 11:18:33.147151 8418 tablet_replica.cc:442] T b0d12efb51814b5e91a1cc2aaf453015 P 1fce593c186f40e88d46e0974b886681: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:18:33.151180 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:33.148254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42627 (local address 127.0.98.3:43039)
0927 11:18:33.148394 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:33.148405 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:33.148460 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:33.148603 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:33.148604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:33.148605 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:18:33.148670 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:33.148671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.149579 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.149580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:33.150321 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:33.150323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.150779 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.150780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:33.150821 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:33.150834 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:18:33.151011 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:18:33.151013 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:18:33.151048 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 11:18:33.151092 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260927 11:18:33.151182 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:33.148312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43039 (local address 127.0.98.3:42627)
0927 11:18:33.148349 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:33.148351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:33.148465 (+ 114us) server_negotiation.cc:407] Connection header received
0927 11:18:33.148511 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:33.148511 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:33.148521 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:33.148602 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:33.148752 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.149494 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.150406 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.150699 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.150774 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:33.150918 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:33.150951 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:33.151003 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:33.151051 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 11:18:33.151076 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":17}
I20260927 11:18:33.153203 8056 raft_consensus.cc:1261] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Refusing update from remote peer 15e42b6a0dbd4000b0f24da694f2b965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:33.153201 8196 raft_consensus.cc:1261] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Refusing update from remote peer 15e42b6a0dbd4000b0f24da694f2b965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:33.153419 8428 consensus_queue.cc:1035] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:33.153537 8428 consensus_queue.cc:1035] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:33.167084 7898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:33.167200 7898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:33.167873 7898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf: Sending DeleteTablet for 3 replicas of tablet 98fe60ba926b4b219ee36f9d950cf0c7
I20260927 11:18:33.168123 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet 98fe60ba926b4b219ee36f9d950cf0c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57768
I20260927 11:18:33.168164 8315 tablet_service.cc:1514] Processing DeleteTablet for tablet 98fe60ba926b4b219ee36f9d950cf0c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58280
I20260927 11:18:33.168213 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet 98fe60ba926b4b219ee36f9d950cf0c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55614
I20260927 11:18:33.168233 8433 tablet_replica.cc:330] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681: stopping tablet replica
I20260927 11:18:33.168258 8435 tablet_replica.cc:330] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965: stopping tablet replica
I20260927 11:18:33.168290 8433 raft_consensus.cc:2229] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:33.168296 8435 raft_consensus.cc:2229] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:33.168299 8434 tablet_replica.cc:330] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a: stopping tablet replica
I20260927 11:18:33.168330 8433 raft_consensus.cc:2258] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:33.168339 8434 raft_consensus.cc:2229] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:33.168346 8435 raft_consensus.cc:2258] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:33.168376 8434 raft_consensus.cc:2258] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:33.168673 8433 ts_tablet_manager.cc:1905] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:33.168675 8434 ts_tablet_manager.cc:1905] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:33.168759 8435 ts_tablet_manager.cc:1905] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:33.170478 8433 ts_tablet_manager.cc:1918] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:33.170478 8434 ts_tablet_manager.cc:1918] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:33.170531 8434 log.cc:1196] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/98fe60ba926b4b219ee36f9d950cf0c7
I20260927 11:18:33.170536 8433 log.cc:1196] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/wal/wals/98fe60ba926b4b219ee36f9d950cf0c7
I20260927 11:18:33.170635 8435 ts_tablet_manager.cc:1918] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:33.170706 8435 log.cc:1196] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/98fe60ba926b4b219ee36f9d950cf0c7
I20260927 11:18:33.170764 8434 ts_tablet_manager.cc:1939] T 98fe60ba926b4b219ee36f9d950cf0c7 P ae8d04c078e3427fbb2eec90a48e0d3a: Deleting consensus metadata
I20260927 11:18:33.170769 8433 ts_tablet_manager.cc:1939] T 98fe60ba926b4b219ee36f9d950cf0c7 P 1fce593c186f40e88d46e0974b886681: Deleting consensus metadata
I20260927 11:18:33.170946 8435 ts_tablet_manager.cc:1939] T 98fe60ba926b4b219ee36f9d950cf0c7 P 15e42b6a0dbd4000b0f24da694f2b965: Deleting consensus metadata
I20260927 11:18:33.171087 7885 catalog_manager.cc:4841] TS 1fce593c186f40e88d46e0974b886681 (127.0.98.1:44187): tablet 98fe60ba926b4b219ee36f9d950cf0c7 (table test-table [id=f2d7816ed56f4de6942b7a8cc49c2e93]) successfully deleted
I20260927 11:18:33.171159 7883 catalog_manager.cc:4841] TS ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437): tablet 98fe60ba926b4b219ee36f9d950cf0c7 (table test-table [id=f2d7816ed56f4de6942b7a8cc49c2e93]) successfully deleted
I20260927 11:18:33.171260 7886 catalog_manager.cc:4841] TS 15e42b6a0dbd4000b0f24da694f2b965 (127.0.98.3:42627): tablet 98fe60ba926b4b219ee36f9d950cf0c7 (table test-table [id=f2d7816ed56f4de6942b7a8cc49c2e93]) successfully deleted
Sep 27 11:18:33 dist-test-slave-lsvg krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 11:18:33 dist-test-slave-lsvg krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:33 dist-test-slave-lsvg krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 11:18:33.192811 7948 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:33.187051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.0.98.62:42561)
0927 11:18:33.187117 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:33.187120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:33.187229 (+ 109us) server_negotiation.cc:407] Connection header received
0927 11:18:33.187266 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:33.187267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:33.187278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:33.187352 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:33.187894 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.188677 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.189512 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.189813 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.189889 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:33.192741 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38318: BlockingRecv error: recv got EOF from 127.0.0.1:38318 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 11:18:33 dist-test-slave-lsvg krb5kdc[7854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:18:33.204566 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:33.193654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.0.98.62:42561)
0927 11:18:33.193696 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:33.193699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:33.197092 (+ 3393us) server_negotiation.cc:407] Connection header received
0927 11:18:33.197196 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:33.197197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:33.197206 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:33.197258 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:33.197822 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.198398 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.199394 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.199577 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.199631 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:33.202141 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:33.202143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:33.202143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:33.202151 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:33.203563 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:33.203927 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:33.203927 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:33.203928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:33.203975 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:33.204160 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:33.204160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:33.204160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:33.204254 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:33.204327 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:33.204450 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 11:18:33.204501 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:18:33.206125 7898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:33.206301 7898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:33.207651 8176 tablet_service.cc:1467] Processing CreateTablet for tablet e3eee83eb90a437689f171844ce0bee9 (DEFAULT_TABLE table=test-table [id=481eea6c158e486992c6e15001ba2643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:33.207656 8315 tablet_service.cc:1467] Processing CreateTablet for tablet e3eee83eb90a437689f171844ce0bee9 (DEFAULT_TABLE table=test-table [id=481eea6c158e486992c6e15001ba2643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:33.207687 8036 tablet_service.cc:1467] Processing CreateTablet for tablet e3eee83eb90a437689f171844ce0bee9 (DEFAULT_TABLE table=test-table [id=481eea6c158e486992c6e15001ba2643]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:33.207788 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3eee83eb90a437689f171844ce0bee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:33.207787 8315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3eee83eb90a437689f171844ce0bee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:33.207839 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3eee83eb90a437689f171844ce0bee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:33.209242 8411 tablet_bootstrap.cc:492] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965: Bootstrap starting.
I20260927 11:18:33.209295 8412 tablet_bootstrap.cc:492] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681: Bootstrap starting.
I20260927 11:18:33.209553 8411 tablet_bootstrap.cc:654] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:33.209604 8412 tablet_bootstrap.cc:654] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:33.210011 8410 tablet_bootstrap.cc:492] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a: Bootstrap starting.
I20260927 11:18:33.210134 8411 tablet_bootstrap.cc:492] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965: No bootstrap required, opened a new log
I20260927 11:18:33.210184 8411 ts_tablet_manager.cc:1397] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:33.210317 8411 raft_consensus.cc:348] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.210398 8411 raft_consensus.cc:374] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:33.210417 8410 tablet_bootstrap.cc:654] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:33.210427 8411 raft_consensus.cc:729] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e42b6a0dbd4000b0f24da694f2b965, State: Initialized, Role: FOLLOWER
I20260927 11:18:33.210472 8411 consensus_queue.cc:260] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.210559 8411 ts_tablet_manager.cc:1428] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:33.210850 8412 tablet_bootstrap.cc:492] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681: No bootstrap required, opened a new log
I20260927 11:18:33.210903 8412 ts_tablet_manager.cc:1397] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:33.211058 8412 raft_consensus.cc:348] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.211140 8412 raft_consensus.cc:374] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:33.211164 8412 raft_consensus.cc:729] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fce593c186f40e88d46e0974b886681, State: Initialized, Role: FOLLOWER
I20260927 11:18:33.211212 8412 consensus_queue.cc:260] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.211300 8412 ts_tablet_manager.cc:1428] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:33.211611 8410 tablet_bootstrap.cc:492] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a: No bootstrap required, opened a new log
I20260927 11:18:33.211687 8410 ts_tablet_manager.cc:1397] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:18:33.211851 8410 raft_consensus.cc:348] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.211920 8410 raft_consensus.cc:374] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:33.211947 8410 raft_consensus.cc:729] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8d04c078e3427fbb2eec90a48e0d3a, State: Initialized, Role: FOLLOWER
I20260927 11:18:33.211987 8410 consensus_queue.cc:260] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.212083 8410 ts_tablet_manager.cc:1428] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:33.566589 8418 raft_consensus.cc:482] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:33.566716 8418 raft_consensus.cc:504] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.567145 8418 leader_election.cc:288] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437), 15e42b6a0dbd4000b0f24da694f2b965 (127.0.98.3:42627)
I20260927 11:18:33.571120 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:33.567346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42627 (local address 127.0.98.1:55623)
0927 11:18:33.567541 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:33.567550 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:33.567612 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:33.567825 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:33.567826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:33.567827 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:33.567933 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:33.567935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.569018 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.569019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:33.569895 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:33.569898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.570857 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.570858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:33.570930 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:33.570935 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:18:33.570988 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 11:18:33.571028 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":61,"threads_started":1}
I20260927 11:18:33.571157 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:33.567573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55623 (local address 127.0.98.3:42627)
0927 11:18:33.567626 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:33.567629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:33.567653 (+ 24us) server_negotiation.cc:407] Connection header received
0927 11:18:33.567717 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:33.567719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:33.567732 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:33.567767 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:33.568017 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.568874 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.570063 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.570754 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.570836 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:33.570852 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:33.571009 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 11:18:33.571055 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:18:33.571367 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3eee83eb90a437689f171844ce0bee9" candidate_uuid: "1fce593c186f40e88d46e0974b886681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e42b6a0dbd4000b0f24da694f2b965" is_pre_election: true
I20260927 11:18:33.571542 8336 raft_consensus.cc:2454] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fce593c186f40e88d46e0974b886681 in term 0.
I20260927 11:18:33.571943 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:33.567335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43437 (local address 127.0.98.1:52719)
0927 11:18:33.567488 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:33.567500 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:33.567612 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:33.567878 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:33.567879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:33.567880 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:33.567933 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:33.567935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.569968 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.569969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:33.570679 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:33.570682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.571665 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.571666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:33.571740 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:33.571745 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:18:33.571797 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 11:18:33.571846 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260927 11:18:33.572081 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:33.567460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52719 (local address 127.0.98.2:43437)
0927 11:18:33.567726 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:33.567728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:33.567735 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:18:33.567763 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:33.567764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:33.567775 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:33.567868 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:33.568999 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.569868 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.570772 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.571500 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.571565 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:33.571903 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:33.571923 (+ 20us) server_negotiation.cc:299] Negotiation successful
0927 11:18:33.571972 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":63,"threads_started":1}
I20260927 11:18:33.572125 7990 leader_election.cc:302] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e42b6a0dbd4000b0f24da694f2b965, 1fce593c186f40e88d46e0974b886681; no voters:
I20260927 11:18:33.572397 8418 raft_consensus.cc:2781] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:33.572450 8418 raft_consensus.cc:482] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:33.572476 8418 raft_consensus.cc:3037] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:33.572479 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3eee83eb90a437689f171844ce0bee9" candidate_uuid: "1fce593c186f40e88d46e0974b886681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" is_pre_election: true
I20260927 11:18:33.572573 8196 raft_consensus.cc:2454] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fce593c186f40e88d46e0974b886681 in term 0.
I20260927 11:18:33.573594 8418 raft_consensus.cc:504] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.573776 8418 leader_election.cc:288] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [CANDIDATE]: Term 1 election: Requested vote from peers ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437), 15e42b6a0dbd4000b0f24da694f2b965 (127.0.98.3:42627)
I20260927 11:18:33.573999 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3eee83eb90a437689f171844ce0bee9" candidate_uuid: "1fce593c186f40e88d46e0974b886681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a"
I20260927 11:18:33.574092 8196 raft_consensus.cc:3037] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:33.574973 8196 raft_consensus.cc:2454] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fce593c186f40e88d46e0974b886681 in term 1.
I20260927 11:18:33.575193 7987 leader_election.cc:302] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fce593c186f40e88d46e0974b886681, ae8d04c078e3427fbb2eec90a48e0d3a; no voters:
I20260927 11:18:33.575305 8418 raft_consensus.cc:2781] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:33.575318 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3eee83eb90a437689f171844ce0bee9" candidate_uuid: "1fce593c186f40e88d46e0974b886681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e42b6a0dbd4000b0f24da694f2b965"
I20260927 11:18:33.575409 8336 raft_consensus.cc:3037] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:33.575464 8418 raft_consensus.cc:686] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [term 1 LEADER]: Becoming Leader. State: Replica: 1fce593c186f40e88d46e0974b886681, State: Running, Role: LEADER
I20260927 11:18:33.575538 8418 consensus_queue.cc:237] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.576310 7898 catalog_manager.cc:5495] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fce593c186f40e88d46e0974b886681 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1fce593c186f40e88d46e0974b886681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } health_report { overall_health: UNKNOWN } } }
I20260927 11:18:33.576385 8336 raft_consensus.cc:2454] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fce593c186f40e88d46e0974b886681 in term 1.
I20260927 11:18:33.654881 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:33.651395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57784 (local address 127.0.98.1:44187)
0927 11:18:33.651647 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:33.651649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:33.651657 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:33.651743 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:33.651744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:33.651774 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:33.651884 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:33.652054 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.652792 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.653979 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.654214 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.654301 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:33.654467 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:33.654614 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:33.654691 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:33.654748 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 11:18:33.654770 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":77,"threads_started":1}
I20260927 11:18:33.655896 8196 raft_consensus.cc:1261] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Refusing update from remote peer 1fce593c186f40e88d46e0974b886681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:33.656016 8336 raft_consensus.cc:1261] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Refusing update from remote peer 1fce593c186f40e88d46e0974b886681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:33.656162 8418 consensus_queue.cc:1035] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:33.656311 8479 consensus_queue.cc:1035] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:33.659464 7899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:33.659579 7899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:33.660250 7899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf: Sending DeleteTablet for 3 replicas of tablet e3eee83eb90a437689f171844ce0bee9
I20260927 11:18:33.660498 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet e3eee83eb90a437689f171844ce0bee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57768
I20260927 11:18:33.660583 8433 tablet_replica.cc:330] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681: stopping tablet replica
I20260927 11:18:33.660597 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet e3eee83eb90a437689f171844ce0bee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55614
I20260927 11:18:33.660626 8433 raft_consensus.cc:2229] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:33.660630 8315 tablet_service.cc:1514] Processing DeleteTablet for tablet e3eee83eb90a437689f171844ce0bee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58280
I20260927 11:18:33.660676 8434 tablet_replica.cc:330] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a: stopping tablet replica
I20260927 11:18:33.660707 8433 raft_consensus.cc:2258] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:33.660722 8435 tablet_replica.cc:330] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965: stopping tablet replica
I20260927 11:18:33.660746 8434 raft_consensus.cc:2229] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:33.660770 8435 raft_consensus.cc:2229] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:33.660786 8434 raft_consensus.cc:2258] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:33.660808 8435 raft_consensus.cc:2258] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:33.661111 8434 ts_tablet_manager.cc:1905] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:33.661209 8435 ts_tablet_manager.cc:1905] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:33.661767 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:33.661904 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:33.662933 8435 ts_tablet_manager.cc:1918] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:33.663020 8435 log.cc:1196] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/e3eee83eb90a437689f171844ce0bee9
I20260927 11:18:33.663273 8435 ts_tablet_manager.cc:1939] T e3eee83eb90a437689f171844ce0bee9 P 15e42b6a0dbd4000b0f24da694f2b965: Deleting consensus metadata
I20260927 11:18:33.663290 8036 tablet_service.cc:1467] Processing CreateTablet for tablet 4072d010c4ee48d8bfca74c832025e25 (DEFAULT_TABLE table=test-table [id=d96a5d748a564adcb2fbba59106304f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:33.663305 8176 tablet_service.cc:1467] Processing CreateTablet for tablet 4072d010c4ee48d8bfca74c832025e25 (DEFAULT_TABLE table=test-table [id=d96a5d748a564adcb2fbba59106304f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:33.663396 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4072d010c4ee48d8bfca74c832025e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:33.663550 8434 ts_tablet_manager.cc:1918] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:33.663546 8315 tablet_service.cc:1467] Processing CreateTablet for tablet 4072d010c4ee48d8bfca74c832025e25 (DEFAULT_TABLE table=test-table [id=d96a5d748a564adcb2fbba59106304f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:33.663599 8434 log.cc:1196] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/e3eee83eb90a437689f171844ce0bee9
I20260927 11:18:33.663637 8315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4072d010c4ee48d8bfca74c832025e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:33.663810 8434 ts_tablet_manager.cc:1939] T e3eee83eb90a437689f171844ce0bee9 P ae8d04c078e3427fbb2eec90a48e0d3a: Deleting consensus metadata
I20260927 11:18:33.664808 8412 tablet_bootstrap.cc:492] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681: Bootstrap starting.
I20260927 11:18:33.664816 8411 tablet_bootstrap.cc:492] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965: Bootstrap starting.
I20260927 11:18:33.664831 8176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4072d010c4ee48d8bfca74c832025e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:33.665126 8411 tablet_bootstrap.cc:654] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:33.665233 8412 tablet_bootstrap.cc:654] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:33.665987 8411 tablet_bootstrap.cc:492] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965: No bootstrap required, opened a new log
I20260927 11:18:33.666031 8411 ts_tablet_manager.cc:1397] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:33.666126 8410 tablet_bootstrap.cc:492] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a: Bootstrap starting.
I20260927 11:18:33.666131 8411 raft_consensus.cc:348] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.666218 8411 raft_consensus.cc:374] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:33.666242 8411 raft_consensus.cc:729] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e42b6a0dbd4000b0f24da694f2b965, State: Initialized, Role: FOLLOWER
I20260927 11:18:33.666287 8411 consensus_queue.cc:260] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.666373 8411 ts_tablet_manager.cc:1428] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:33.666471 7886 catalog_manager.cc:4841] TS 15e42b6a0dbd4000b0f24da694f2b965 (127.0.98.3:42627): tablet e3eee83eb90a437689f171844ce0bee9 (table test-table [id=481eea6c158e486992c6e15001ba2643]) successfully deleted
I20260927 11:18:33.666522 8410 tablet_bootstrap.cc:654] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:33.667084 8410 tablet_bootstrap.cc:492] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a: No bootstrap required, opened a new log
I20260927 11:18:33.667137 8410 ts_tablet_manager.cc:1397] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:33.667186 7883 catalog_manager.cc:4841] TS ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437): tablet e3eee83eb90a437689f171844ce0bee9 (table test-table [id=481eea6c158e486992c6e15001ba2643]) successfully deleted
I20260927 11:18:33.667294 8410 raft_consensus.cc:348] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.667367 8410 raft_consensus.cc:374] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:33.667388 8410 raft_consensus.cc:729] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8d04c078e3427fbb2eec90a48e0d3a, State: Initialized, Role: FOLLOWER
I20260927 11:18:33.667385 8433 ts_tablet_manager.cc:1905] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:33.667433 8410 consensus_queue.cc:260] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.667541 8410 ts_tablet_manager.cc:1428] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:33.668928 8433 ts_tablet_manager.cc:1918] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:33.669004 8433 log.cc:1196] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-0/wal/wals/e3eee83eb90a437689f171844ce0bee9
I20260927 11:18:33.669242 8433 ts_tablet_manager.cc:1939] T e3eee83eb90a437689f171844ce0bee9 P 1fce593c186f40e88d46e0974b886681: Deleting consensus metadata
I20260927 11:18:33.669552 7885 catalog_manager.cc:4841] TS 1fce593c186f40e88d46e0974b886681 (127.0.98.1:44187): tablet e3eee83eb90a437689f171844ce0bee9 (table test-table [id=481eea6c158e486992c6e15001ba2643]) successfully deleted
I20260927 11:18:33.669610 8412 tablet_bootstrap.cc:492] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681: No bootstrap required, opened a new log
I20260927 11:18:33.669659 8412 ts_tablet_manager.cc:1397] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260927 11:18:33.669804 8412 raft_consensus.cc:348] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.669859 8412 raft_consensus.cc:374] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:33.669873 8412 raft_consensus.cc:729] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fce593c186f40e88d46e0974b886681, State: Initialized, Role: FOLLOWER
I20260927 11:18:33.669925 8412 consensus_queue.cc:260] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.670034 8412 ts_tablet_manager.cc:1428] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:33.728164 8417 raft_consensus.cc:482] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:33.728271 8417 raft_consensus.cc:504] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.728433 8417 leader_election.cc:288] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fce593c186f40e88d46e0974b886681 (127.0.98.1:44187), ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437)
I20260927 11:18:33.728655 8055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4072d010c4ee48d8bfca74c832025e25" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fce593c186f40e88d46e0974b886681" is_pre_election: true
I20260927 11:18:33.728682 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4072d010c4ee48d8bfca74c832025e25" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" is_pre_election: true
I20260927 11:18:33.728771 8055 raft_consensus.cc:2454] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 0.
I20260927 11:18:33.728793 8196 raft_consensus.cc:2454] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 0.
I20260927 11:18:33.728940 8269 leader_election.cc:302] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e42b6a0dbd4000b0f24da694f2b965, 1fce593c186f40e88d46e0974b886681; no voters:
I20260927 11:18:33.729074 8417 raft_consensus.cc:2781] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:33.729110 8417 raft_consensus.cc:482] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:33.729130 8417 raft_consensus.cc:3037] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:33.729914 8417 raft_consensus.cc:504] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.730044 8417 leader_election.cc:288] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 election: Requested vote from peers 1fce593c186f40e88d46e0974b886681 (127.0.98.1:44187), ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437)
I20260927 11:18:33.730172 8055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4072d010c4ee48d8bfca74c832025e25" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fce593c186f40e88d46e0974b886681"
I20260927 11:18:33.730248 8055 raft_consensus.cc:3037] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:33.730229 8196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4072d010c4ee48d8bfca74c832025e25" candidate_uuid: "15e42b6a0dbd4000b0f24da694f2b965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a"
I20260927 11:18:33.730309 8196 raft_consensus.cc:3037] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:33.730866 8055 raft_consensus.cc:2454] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 1.
I20260927 11:18:33.731066 8269 leader_election.cc:302] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e42b6a0dbd4000b0f24da694f2b965, 1fce593c186f40e88d46e0974b886681; no voters:
I20260927 11:18:33.731119 8196 raft_consensus.cc:2454] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e42b6a0dbd4000b0f24da694f2b965 in term 1.
I20260927 11:18:33.731146 8417 raft_consensus.cc:2781] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:33.731261 8417 raft_consensus.cc:686] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 LEADER]: Becoming Leader. State: Replica: 15e42b6a0dbd4000b0f24da694f2b965, State: Running, Role: LEADER
I20260927 11:18:33.731316 8417 consensus_queue.cc:237] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } }
I20260927 11:18:33.731879 7898 catalog_manager.cc:5495] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15e42b6a0dbd4000b0f24da694f2b965 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "15e42b6a0dbd4000b0f24da694f2b965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e42b6a0dbd4000b0f24da694f2b965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42627 } health_report { overall_health: HEALTHY } } }
I20260927 11:18:33.745796 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:33.743530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.0.98.3:42627)
0927 11:18:33.743574 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:33.743577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:33.743586 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:33.743659 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:33.743659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:33.743671 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:33.743725 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:33.743833 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.744311 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.745038 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:33.745305 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:33.745386 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:33.745531 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:33.745571 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:33.745632 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:33.745701 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 11:18:33.745719 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:18:33.747750 8055 raft_consensus.cc:1261] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Refusing update from remote peer 15e42b6a0dbd4000b0f24da694f2b965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:33.748023 8196 raft_consensus.cc:1261] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Refusing update from remote peer 15e42b6a0dbd4000b0f24da694f2b965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:33.748003 8491 consensus_queue.cc:1035] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fce593c186f40e88d46e0974b886681" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:33.748224 8417 consensus_queue.cc:1035] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8d04c078e3427fbb2eec90a48e0d3a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:33.757614 7898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:33.757704 7898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:33.758327 7898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25acfc3c658b4b33894526e0dfda20bf: Sending DeleteTablet for 3 replicas of tablet 4072d010c4ee48d8bfca74c832025e25
I20260927 11:18:33.758669 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet 4072d010c4ee48d8bfca74c832025e25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57768
I20260927 11:18:33.758775 8176 tablet_service.cc:1514] Processing DeleteTablet for tablet 4072d010c4ee48d8bfca74c832025e25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55614
I20260927 11:18:33.758788 8433 tablet_replica.cc:330] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681: stopping tablet replica
I20260927 11:18:33.758849 8433 raft_consensus.cc:2229] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:33.758852 8434 tablet_replica.cc:330] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a: stopping tablet replica
I20260927 11:18:33.758893 8433 raft_consensus.cc:2258] T 4072d010c4ee48d8bfca74c832025e25 P 1fce593c186f40e88d46e0974b886681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:33.758896 8434 raft_consensus.cc:2229] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:33.758905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 7971
I20260927 11:18:33.758927 8434 raft_consensus.cc:2258] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:33.760190 8316 tablet_service.cc:1514] Processing DeleteTablet for tablet 4072d010c4ee48d8bfca74c832025e25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58280
I20260927 11:18:33.760284 8435 tablet_replica.cc:330] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965: stopping tablet replica
I20260927 11:18:33.760351 8435 raft_consensus.cc:2229] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:33.760416 8435 raft_consensus.cc:2258] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:33.760423 8434 ts_tablet_manager.cc:1905] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:33.760921 8435 ts_tablet_manager.cc:1905] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:33.762158 8434 ts_tablet_manager.cc:1918] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:33.762253 8434 log.cc:1196] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/4072d010c4ee48d8bfca74c832025e25
I20260927 11:18:33.762468 8434 ts_tablet_manager.cc:1939] T 4072d010c4ee48d8bfca74c832025e25 P ae8d04c078e3427fbb2eec90a48e0d3a: Deleting consensus metadata
I20260927 11:18:33.762614 8435 ts_tablet_manager.cc:1918] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:18:33.762686 8435 log.cc:1196] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/4072d010c4ee48d8bfca74c832025e25
I20260927 11:18:33.762801 7883 catalog_manager.cc:4841] TS ae8d04c078e3427fbb2eec90a48e0d3a (127.0.98.2:43437): tablet 4072d010c4ee48d8bfca74c832025e25 (table test-table [id=d96a5d748a564adcb2fbba59106304f6]) successfully deleted
I20260927 11:18:33.762899 8435 ts_tablet_manager.cc:1939] T 4072d010c4ee48d8bfca74c832025e25 P 15e42b6a0dbd4000b0f24da694f2b965: Deleting consensus metadata
I20260927 11:18:33.763338 7886 catalog_manager.cc:4841] TS 15e42b6a0dbd4000b0f24da694f2b965 (127.0.98.3:42627): tablet 4072d010c4ee48d8bfca74c832025e25 (table test-table [id=d96a5d748a564adcb2fbba59106304f6]) successfully deleted
W20260927 11:18:33.766301 7885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44187 (error 108)
I20260927 11:18:33.766508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 8111
I20260927 11:18:33.767771 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:33.766677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44187 (local address 127.0.0.1:57788)
0927 11:18:33.767671 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:33.767723 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":975,"thread_start_us":45,"threads_started":1}
W20260927 11:18:33.767885 7885 catalog_manager.cc:4584] TS 1fce593c186f40e88d46e0974b886681 (127.0.98.1:44187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4072d010c4ee48d8bfca74c832025e25: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44187: connect: Connection refused (error 111)
I20260927 11:18:33.772099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 8251
I20260927 11:18:33.777987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 7867
2026-09-27T11:18:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3070 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:33 dist-test-slave-lsvg krb5kdc[8501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:33 dist-test-slave-lsvg krb5kdc[8501](info): set up 2 sockets
Sep 27 11:18:33 dist-test-slave-lsvg krb5kdc[8501](info): commencing operation
krb5kdc: starting...
W20260927 11:18:34.813872 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:34 dist-test-slave-lsvg krb5kdc[8501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:18:34.931927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38811
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:35.014686 8514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:35.014881 8514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:35.014902 8514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:35.015971 8514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:35.016003 8514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:35.016017 8514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:35.016031 8514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:35.016042 8514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:35.017501 8514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:35.017688 8514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:35.017856 8514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:35.022549 8519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:35.022549 8522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:35.022593 8514 server_base.cc:1034] running on GCE node
W20260927 11:18:35.022572 8520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:35.023072 8514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:35.023361 8514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:35.024539 8514 hybrid_clock.cc:648] HybridClock initialized: now 1790507915024520 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T11:18:35Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1250 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:35 dist-test-slave-lsvg krb5kdc[8527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:35 dist-test-slave-lsvg krb5kdc[8527](info): set up 2 sockets
Sep 27 11:18:35 dist-test-slave-lsvg krb5kdc[8527](info): commencing operation
krb5kdc: starting...
W20260927 11:18:36.077870 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:36 dist-test-slave-lsvg krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:18:36.200605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46403
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:36.281420 8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:36.281634 8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:36.281658 8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:36.282847 8540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:36.282894 8540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:36.282909 8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:36.282922 8540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:36.282979 8540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:36.284510 8540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:36.284758 8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:36.284989 8540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:36.289906 8548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:36.289983 8546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:36.289975 8545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:36.290120 8540 server_base.cc:1034] running on GCE node
I20260927 11:18:36.290380 8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:36.290684 8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:36.291852 8540 hybrid_clock.cc:648] HybridClock initialized: now 1790507916291831 us; error 41 us; skew 500 ppm
Sep 27 11:18:36 dist-test-slave-lsvg krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:36.294425 8540 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:18:36.294891 8540 webserver.cc:460] Webserver started at http://127.0.98.62:45123/ using document root <none> and password file <none>
I20260927 11:18:36.295097 8540 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:36.295142 8540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:36.295233 8540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:36.296250 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "32c40ee19d75429aa3e383375679e4a2"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "3d4fa6af132cb2ee928b01c4fe8a2305"
server_key_iv: "80ead9ffeb5aaba62c37e0404feb2928"
server_key_version: "encryptionkey@0"
I20260927 11:18:36.296554 8540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "32c40ee19d75429aa3e383375679e4a2"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "3d4fa6af132cb2ee928b01c4fe8a2305"
server_key_iv: "80ead9ffeb5aaba62c37e0404feb2928"
server_key_version: "encryptionkey@0"
I20260927 11:18:36.297891 8540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:36.298556 8555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:36.298748 8540 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:36.298837 8540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "32c40ee19d75429aa3e383375679e4a2"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "3d4fa6af132cb2ee928b01c4fe8a2305"
server_key_iv: "80ead9ffeb5aaba62c37e0404feb2928"
server_key_version: "encryptionkey@0"
I20260927 11:18:36.298906 8540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:36.316571 8540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:36.318677 8540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:36.318837 8540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:36.322842 8540 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46403
I20260927 11:18:36.322880 8607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46403 every 8 connection(s)
I20260927 11:18:36.323415 8540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:36.324097 8608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:36.325301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 8540
I20260927 11:18:36.325409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:36.325603 392 external_mini_cluster.cc:1428] Setting key 17658c85390698c4b8a12beed4a0092f
I20260927 11:18:36.327080 8608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2: Bootstrap starting.
I20260927 11:18:36.327672 8608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:36.328006 8608 log.cc:824] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:36.328785 8608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2: No bootstrap required, opened a new log
I20260927 11:18:36.331068 8608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c40ee19d75429aa3e383375679e4a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260927 11:18:36.331243 8608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:36.331272 8608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c40ee19d75429aa3e383375679e4a2, State: Initialized, Role: FOLLOWER
I20260927 11:18:36.331426 8608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c40ee19d75429aa3e383375679e4a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260927 11:18:36.331506 8608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:36.331528 8608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:36.331602 8608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:36.332131 8608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c40ee19d75429aa3e383375679e4a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260927 11:18:36.332250 8608 leader_election.cc:302] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32c40ee19d75429aa3e383375679e4a2; no voters:
I20260927 11:18:36.332422 8608 leader_election.cc:288] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:36.332491 8613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:36.332640 8613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 1 LEADER]: Becoming Leader. State: Replica: 32c40ee19d75429aa3e383375679e4a2, State: Running, Role: LEADER
I20260927 11:18:36.332684 8608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:36.332737 8613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c40ee19d75429aa3e383375679e4a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
Sep 27 11:18:36 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:36.333065 8615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32c40ee19d75429aa3e383375679e4a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c40ee19d75429aa3e383375679e4a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } } }
I20260927 11:18:36.333180 8615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:36.333038 8613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c40ee19d75429aa3e383375679e4a2. Latest consensus state: current_term: 1 leader_uuid: "32c40ee19d75429aa3e383375679e4a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c40ee19d75429aa3e383375679e4a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } } }
I20260927 11:18:36.333372 8613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:36.333489 8620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:36.334146 8620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:36.335847 8620 catalog_manager.cc:1347] Generated new cluster ID: 0c14172e07804cb2bc03a87cb6c0cb0c
I20260927 11:18:36.335898 8620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:36.337422 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:36.326148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.0.98.62:46403)
0927 11:18:36.326339 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:36.326348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:36.326379 (+ 31us) server_negotiation.cc:407] Connection header received
0927 11:18:36.326890 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:36.326898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:36.326960 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:36.327130 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:36.327768 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.329134 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.329867 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.330573 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.330683 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:36.333520 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:36.333524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:36.333525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:36.333536 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:36.335736 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:36.336158 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:36.336161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:36.336161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:36.336256 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:36.336489 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:36.336490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:36.336491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:36.336806 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:36.336962 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:36.337092 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 11:18:36.337233 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260927 11:18:36.345896 8620 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:36.346604 8620 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:36.364013 8620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2: Generated new TSK 0
I20260927 11:18:36.364246 8620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:18:36.390383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:36.471915 8634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:36.472117 8634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:36.472141 8634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:36.473331 8634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:36.473373 8634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:36.473433 8634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:36.474912 8634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:36.475154 8634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:36.475399 8634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:36.480185 8639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:36.480137 8642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:36.480134 8640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:36.480631 8634 server_base.cc:1034] running on GCE node
I20260927 11:18:36.480822 8634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:36.481009 8634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:36.482196 8634 hybrid_clock.cc:648] HybridClock initialized: now 1790507916482178 us; error 28 us; skew 500 ppm
Sep 27 11:18:36 dist-test-slave-lsvg krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:36.484709 8634 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:18:36.485179 8634 webserver.cc:460] Webserver started at http://127.0.98.1:42147/ using document root <none> and password file <none>
I20260927 11:18:36.485369 8634 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:36.485409 8634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:36.485495 8634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:36.486454 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "46c3b52d86994a2c9ecefb556c2c6eed"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "ed0b7c56ad400b768c2363bbd0930436"
server_key_iv: "8985b3c3d2e24d4e6b73dfac640323a5"
server_key_version: "encryptionkey@0"
I20260927 11:18:36.486749 8634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "46c3b52d86994a2c9ecefb556c2c6eed"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "ed0b7c56ad400b768c2363bbd0930436"
server_key_iv: "8985b3c3d2e24d4e6b73dfac640323a5"
server_key_version: "encryptionkey@0"
I20260927 11:18:36.487931 8634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:36.488574 8649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:36.488749 8634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:36.488845 8634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "46c3b52d86994a2c9ecefb556c2c6eed"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "ed0b7c56ad400b768c2363bbd0930436"
server_key_iv: "8985b3c3d2e24d4e6b73dfac640323a5"
server_key_version: "encryptionkey@0"
I20260927 11:18:36.488924 8634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:36.502959 8634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:36.504761 8634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:36.504916 8634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:36.505199 8634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:36.505512 8634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:36.505574 8634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:36.505617 8634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:36.505646 8634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:36.511229 8634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38187
I20260927 11:18:36.511300 8762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38187 every 8 connection(s)
I20260927 11:18:36.511634 8634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:18:36.515348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 8634
I20260927 11:18:36.515472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:36.515648 392 external_mini_cluster.cc:1428] Setting key c721567c876a215ca6094991fab92e1c
Sep 27 11:18:36 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:36.521441 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:36.512345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51265 (local address 127.0.98.62:46403)
0927 11:18:36.512408 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:36.512410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:36.512628 (+ 218us) server_negotiation.cc:407] Connection header received
0927 11:18:36.513671 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:36.513673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:36.513683 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:36.513743 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:36.514322 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.514802 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.515902 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.516190 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.516257 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:36.518643 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:36.518645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:36.518645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:36.518653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:36.520008 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:36.520419 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:36.520421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:36.520421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:36.520477 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:36.520735 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:36.520736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:36.520736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:36.520854 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:36.520931 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:36.521271 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 11:18:36.521354 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 11:18:36.521639 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:36.512326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:51265)
0927 11:18:36.512488 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:36.512529 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:36.513557 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:36.513801 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:36.513804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:36.514055 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:36.514242 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:36.514246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.514878 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.514879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:36.515789 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:36.515791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.516265 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.516266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:36.516435 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:36.516936 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:36.516946 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:36.518539 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:36.520115 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:36.520121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:36.520123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:36.520343 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:36.520560 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:36.520561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:36.520561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:36.520667 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:36.520922 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:36.520927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:36.521122 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 11:18:36.521234 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 11:18:36.521390 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260927 11:18:36.522105 8763 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46403
I20260927 11:18:36.522228 8763 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:36.522428 8763 heartbeater.cc:511] Master 127.0.98.62:46403 requested a full tablet report, sending...
I20260927 11:18:36.522938 8572 ts_manager.cc:194] Registered new tserver with Master: 46c3b52d86994a2c9ecefb556c2c6eed (127.0.98.1:38187)
I20260927 11:18:36.523559 8572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51265
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:18:36.540107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:36.621289 8768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:36.621498 8768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:36.621523 8768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:36.622635 8768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:36.622669 8768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:36.622695 8768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:36.624178 8768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:36.624369 8768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:36.624550 8768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:36.629413 8774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:36.629412 8776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:36.629509 8773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:36.629544 8768 server_base.cc:1034] running on GCE node
I20260927 11:18:36.629889 8768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:36.630133 8768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:36.631301 8768 hybrid_clock.cc:648] HybridClock initialized: now 1790507916631284 us; error 37 us; skew 500 ppm
Sep 27 11:18:36 dist-test-slave-lsvg krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:36.633996 8768 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:18:36.634428 8768 webserver.cc:460] Webserver started at http://127.0.98.2:39397/ using document root <none> and password file <none>
I20260927 11:18:36.634625 8768 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:36.634665 8768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:36.634752 8768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:36.635828 8768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "72282ba6a24245c5b548e56a4cc28a12"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "de25b095e5ad7907136db6733f08895a"
server_key_iv: "cbc2501ab727bcac40558cda40d1dc15"
server_key_version: "encryptionkey@0"
I20260927 11:18:36.636122 8768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "72282ba6a24245c5b548e56a4cc28a12"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "de25b095e5ad7907136db6733f08895a"
server_key_iv: "cbc2501ab727bcac40558cda40d1dc15"
server_key_version: "encryptionkey@0"
I20260927 11:18:36.637357 8768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:36.638029 8783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:36.638204 8768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:36.638306 8768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "72282ba6a24245c5b548e56a4cc28a12"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "de25b095e5ad7907136db6733f08895a"
server_key_iv: "cbc2501ab727bcac40558cda40d1dc15"
server_key_version: "encryptionkey@0"
I20260927 11:18:36.638381 8768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:36.655781 8768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:36.657982 8768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:36.658175 8768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:36.658419 8768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:36.658728 8768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:36.658780 8768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:36.658821 8768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:36.658849 8768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:36.664839 8768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42145
I20260927 11:18:36.664918 8896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42145 every 8 connection(s)
I20260927 11:18:36.665285 8768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:18:36 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:36.674921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 8768
I20260927 11:18:36.675033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:36.675060 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:36.666320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51985 (local address 127.0.98.62:46403)
0927 11:18:36.666372 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:36.666374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:36.667196 (+ 822us) server_negotiation.cc:407] Connection header received
0927 11:18:36.668133 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:36.668134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:36.668144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:36.668196 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:36.668768 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.669326 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.670294 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.670476 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.670529 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:36.672563 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:36.672564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:36.672565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:36.672571 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:36.673819 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:36.674181 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:36.674182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:36.674182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:36.674231 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:36.674438 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:36.674439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:36.674439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:36.674534 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:36.674585 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:36.674854 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 11:18:36.674938 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:18:36.675230 392 external_mini_cluster.cc:1428] Setting key f40f9abfcf87532d39479c591522a370
I20260927 11:18:36.675228 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:36.666308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:51985)
0927 11:18:36.667040 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:36.667091 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:36.668041 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:36.668250 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:36.668253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:36.668502 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:36.668698 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:36.668702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.669398 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.669399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:36.670214 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:36.670216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.670550 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.670550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:36.670665 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:36.671132 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:36.671140 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:36.672471 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:36.673888 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:36.673894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:36.673896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:36.674114 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:36.674290 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:36.674290 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:36.674291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:36.674385 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:36.674599 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:36.674602 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:36.674729 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 11:18:36.674811 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 11:18:36.674957 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":82,"threads_started":1}
I20260927 11:18:36.675758 8897 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46403
I20260927 11:18:36.675855 8897 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:36.676033 8897 heartbeater.cc:511] Master 127.0.98.62:46403 requested a full tablet report, sending...
I20260927 11:18:36.676466 8571 ts_manager.cc:194] Registered new tserver with Master: 72282ba6a24245c5b548e56a4cc28a12 (127.0.98.2:42145)
I20260927 11:18:36.676929 8571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51985
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:18:36.700347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:36.780548 8902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:36.780764 8902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:36.780789 8902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:36.781934 8902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:36.781971 8902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:36.781997 8902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:36.783447 8902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:36.783639 8902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:36.783821 8902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:36.789155 8910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:36.789186 8907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:36.789158 8908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:36.789429 8902 server_base.cc:1034] running on GCE node
I20260927 11:18:36.789680 8902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:36.789939 8902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:36.791132 8902 hybrid_clock.cc:648] HybridClock initialized: now 1790507916791095 us; error 60 us; skew 500 ppm
Sep 27 11:18:36 dist-test-slave-lsvg krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:36.794020 8902 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:18:36.794524 8902 webserver.cc:460] Webserver started at http://127.0.98.3:42327/ using document root <none> and password file <none>
I20260927 11:18:36.794752 8902 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:36.794833 8902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:36.794992 8902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:36.796101 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "87b15df824214203b68bb1395f802603"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "901001f87dc48e923b627652ea65f018"
server_key_iv: "0537d26cae126f5b287f859f52e6308d"
server_key_version: "encryptionkey@0"
I20260927 11:18:36.796471 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "87b15df824214203b68bb1395f802603"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "901001f87dc48e923b627652ea65f018"
server_key_iv: "0537d26cae126f5b287f859f52e6308d"
server_key_version: "encryptionkey@0"
I20260927 11:18:36.797940 8902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 11:18:36.798803 8917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:36.798991 8902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:36.799108 8902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "87b15df824214203b68bb1395f802603"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "901001f87dc48e923b627652ea65f018"
server_key_iv: "0537d26cae126f5b287f859f52e6308d"
server_key_version: "encryptionkey@0"
I20260927 11:18:36.799185 8902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:36.821101 8902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:36.822960 8902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:36.823132 8902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:36.823376 8902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:36.823683 8902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:36.823737 8902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:36.823781 8902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:36.823808 8902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:36.829780 8902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43351
I20260927 11:18:36.829823 9030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43351 every 8 connection(s)
I20260927 11:18:36.830150 8902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:36.835513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 8902
I20260927 11:18:36.835615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:36.835793 392 external_mini_cluster.cc:1428] Setting key ba3a2bd257eea4b811485c78c04fda32
Sep 27 11:18:36 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:36.839558 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:36.830866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41289 (local address 127.0.98.62:46403)
0927 11:18:36.830954 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:36.830956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:36.831132 (+ 176us) server_negotiation.cc:407] Connection header received
0927 11:18:36.832077 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:36.832078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:36.832088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:36.832150 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:36.832858 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.833483 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.834546 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.834723 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.834797 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:36.837244 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:36.837246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:36.837246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:36.837252 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:36.838486 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:36.838826 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:36.838827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:36.838827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:36.838875 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:36.839088 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:36.839088 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:36.839089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:36.839187 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:36.839238 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:36.839441 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 11:18:36.839496 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 11:18:36.839795 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:36.830830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:41289)
0927 11:18:36.831008 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:36.831045 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:36.831975 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:36.832214 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:36.832217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:36.832562 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:36.832763 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:36.832769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.833568 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.833569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:36.834468 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:36.834471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:36.834809 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:36.834810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:36.834907 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:36.835469 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:36.835478 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:36.837181 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:36.838554 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:36.838559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:36.838561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:36.838762 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:36.838930 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:36.838930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:36.838930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:36.839021 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:36.839240 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:36.839244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:36.839351 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 11:18:36.839430 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 11:18:36.839543 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260927 11:18:36.840233 9031 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46403
I20260927 11:18:36.840312 9031 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:36.840497 9031 heartbeater.cc:511] Master 127.0.98.62:46403 requested a full tablet report, sending...
I20260927 11:18:36.840883 8571 ts_manager.cc:194] Registered new tserver with Master: 87b15df824214203b68bb1395f802603 (127.0.98.3:43351)
I20260927 11:18:36.841301 8571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41289
I20260927 11:18:36.847795 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 11:18:36.920230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 8634
I20260927 11:18:36.924736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 8768
I20260927 11:18:36.928987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 8902
I20260927 11:18:36.933424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 8540
I20260927 11:18:36.938362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46403
--webserver_interface=127.0.98.62
--webserver_port=45123
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:37.020077 9040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:37.020331 9040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:37.020370 9040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:37.021636 9040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:37.021703 9040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:37.021729 9040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:37.021761 9040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:37.021785 9040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:37.023312 9040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:37.023595 9040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:37.023823 9040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:37.028823 9048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:37.028891 9040 server_base.cc:1034] running on GCE node
W20260927 11:18:37.028817 9046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:37.029017 9045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:37.029300 9040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:37.029609 9040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:37.030767 9040 hybrid_clock.cc:648] HybridClock initialized: now 1790507917030749 us; error 41 us; skew 500 ppm
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:37.033399 9040 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:18:37.033897 9040 webserver.cc:460] Webserver started at http://127.0.98.62:45123/ using document root <none> and password file <none>
I20260927 11:18:37.034096 9040 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:37.034152 9040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:37.035459 9040 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:37.036082 9055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:37.036405 9040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:37.036504 9040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "32c40ee19d75429aa3e383375679e4a2"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "3d4fa6af132cb2ee928b01c4fe8a2305"
server_key_iv: "80ead9ffeb5aaba62c37e0404feb2928"
server_key_version: "encryptionkey@0"
I20260927 11:18:37.036870 9040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:37.063154 9040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:37.065641 9040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:37.065805 9040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:37.069898 9040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46403
I20260927 11:18:37.069936 9107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46403 every 8 connection(s)
I20260927 11:18:37.070501 9040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:37.071761 9108 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:18:37.072461 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2: Bootstrap starting.
I20260927 11:18:37.072974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 9040
I20260927 11:18:37.073320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38187
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46403
--webserver_port=42147
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:18:37.081280 9108 log.cc:824] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:37.083009 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:18:37.083325 9108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2: Bootstrap complete.
I20260927 11:18:37.085664 9108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c40ee19d75429aa3e383375679e4a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260927 11:18:37.085892 9108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c40ee19d75429aa3e383375679e4a2, State: Initialized, Role: FOLLOWER
I20260927 11:18:37.086073 9108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c40ee19d75429aa3e383375679e4a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260927 11:18:37.086149 9108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:37.086197 9108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:37.086246 9108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:18:37.087335 9108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c40ee19d75429aa3e383375679e4a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260927 11:18:37.087425 9108 leader_election.cc:302] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32c40ee19d75429aa3e383375679e4a2; no voters:
I20260927 11:18:37.087630 9108 leader_election.cc:288] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 11:18:37.087697 9112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 11:18:37.087900 9108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:37.087914 9112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [term 2 LEADER]: Becoming Leader. State: Replica: 32c40ee19d75429aa3e383375679e4a2, State: Running, Role: LEADER
I20260927 11:18:37.088140 9112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c40ee19d75429aa3e383375679e4a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260927 11:18:37.088467 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "32c40ee19d75429aa3e383375679e4a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c40ee19d75429aa3e383375679e4a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } } }
I20260927 11:18:37.088617 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:37.088797 9114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c40ee19d75429aa3e383375679e4a2. Latest consensus state: current_term: 2 leader_uuid: "32c40ee19d75429aa3e383375679e4a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c40ee19d75429aa3e383375679e4a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } } }
I20260927 11:18:37.088884 9114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:37.089174 9121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:37.089771 9121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:37.089910 9121 catalog_manager.cc:1259] Loaded cluster ID: 0c14172e07804cb2bc03a87cb6c0cb0c
I20260927 11:18:37.089950 9121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:37.090749 9121 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:37.090936 9121 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 32c40ee19d75429aa3e383375679e4a2: Loaded TSK: 0
I20260927 11:18:37.091146 9121 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 11:18:37.156931 9110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:37.157253 9110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:37.157301 9110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:37.158489 9110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:37.158556 9110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:37.158602 9110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:37.160044 9110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42147
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:37.160306 9110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:37.160548 9110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:37.166462 9134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:37.166489 9135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:37.166591 9137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:37.166872 9110 server_base.cc:1034] running on GCE node
I20260927 11:18:37.167091 9110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:37.167330 9110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:37.168494 9110 hybrid_clock.cc:648] HybridClock initialized: now 1790507917168482 us; error 31 us; skew 500 ppm
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507917, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:37.171123 9110 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 11:18:37.171645 9110 webserver.cc:460] Webserver started at http://127.0.98.1:42147/ using document root <none> and password file <none>
I20260927 11:18:37.171885 9110 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:37.171967 9110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:37.173332 9110 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:37.173962 9144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:37.174142 9110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:18:37.174245 9110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "46c3b52d86994a2c9ecefb556c2c6eed"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "ed0b7c56ad400b768c2363bbd0930436"
server_key_iv: "8985b3c3d2e24d4e6b73dfac640323a5"
server_key_version: "encryptionkey@0"
I20260927 11:18:37.174594 9110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:37.191331 9110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:37.193675 9110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:37.193825 9110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:37.194123 9110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:37.194509 9110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:37.194546 9110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:37.194571 9110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:37.194583 9110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:37.201845 9110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38187
I20260927 11:18:37.201932 9257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38187 every 8 connection(s)
I20260927 11:18:37.202278 9110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:18:37.208630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 9110
I20260927 11:18:37.208958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42145
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46403
--webserver_port=39397
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:37.211874 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:37.203049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:60791)
0927 11:18:37.203228 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:37.203272 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:37.203737 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:37.204224 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:37.204227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:37.204790 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:37.204995 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.205001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.206665 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.206666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.207478 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.207480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.208284 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.208285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.208393 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.209020 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:37.209032 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:37.211645 (+ 2613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
W20260927 11:18:37.212034 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:37.203145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60791 (local address 127.0.98.62:46403)
0927 11:18:37.203359 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:37.203369 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:37.203406 (+ 37us) server_negotiation.cc:407] Connection header received
0927 11:18:37.203961 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:37.203968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:37.204025 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:37.204183 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:37.205175 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.206556 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.207568 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.208192 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.208299 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.211810 (+ 3511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60791: BlockingRecv error: recv got EOF from 127.0.98.1:60791 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":93,"threads_started":1}
W20260927 11:18:37.212430 9258 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46403 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:46403: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:37.218400 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:37.212676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:40395)
0927 11:18:37.212721 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:37.212732 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:37.212769 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:37.212923 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:37.212924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:37.213122 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:37.213177 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.213179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.214193 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.214194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.215027 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.215029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.215577 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.215578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.215663 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.216204 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:37.216209 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:37.218326 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 11:18:37.218637 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:37.212710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40395 (local address 127.0.98.62:46403)
0927 11:18:37.212751 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:37.212753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:37.212763 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:37.212824 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:37.212824 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:37.212837 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:37.212908 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:37.213272 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.214099 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.215133 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.215482 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.215564 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.218452 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40395: BlockingRecv error: recv got EOF from 127.0.98.1:40395 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:37.224123 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:37.218706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:54521)
0927 11:18:37.218842 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:37.218851 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:37.218884 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:37.219169 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:37.219170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:37.219309 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:37.219343 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.219344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.220285 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.220286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.221115 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.221117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.221578 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.221579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.221644 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.222141 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:37.222149 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:37.224046 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
W20260927 11:18:37.224208 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:37.218814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54521 (local address 127.0.98.62:46403)
0927 11:18:37.219047 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:37.219049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:37.219058 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:37.219081 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:37.219082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:37.219093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:37.219157 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:37.219429 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.220206 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.221193 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.221498 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.221566 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.224149 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54521: BlockingRecv error: recv got EOF from 127.0.98.1:54521 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":99,"threads_started":1}
W20260927 11:18:37.224448 9258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 11:18:37.292906 9262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:37.293197 9262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:37.293241 9262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:37.294456 9262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:37.294526 9262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:37.294570 9262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:37.296011 9262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39397
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:37.296309 9262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:37.296546 9262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:37.301625 9269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:37.301618 9270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:37.301618 9272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:37.301798 9262 server_base.cc:1034] running on GCE node
I20260927 11:18:37.302071 9262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:37.302330 9262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:37.303489 9262 hybrid_clock.cc:648] HybridClock initialized: now 1790507917303464 us; error 39 us; skew 500 ppm
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507917, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:37.306001 9262 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 11:18:37.306479 9262 webserver.cc:460] Webserver started at http://127.0.98.2:39397/ using document root <none> and password file <none>
I20260927 11:18:37.306718 9262 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:37.306798 9262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:37.308113 9262 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:37.308708 9279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:37.308864 9262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:37.308962 9262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "72282ba6a24245c5b548e56a4cc28a12"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "de25b095e5ad7907136db6733f08895a"
server_key_iv: "cbc2501ab727bcac40558cda40d1dc15"
server_key_version: "encryptionkey@0"
I20260927 11:18:37.309373 9262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:37.331624 9262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:37.334107 9262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:37.334299 9262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:37.334556 9262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:37.334870 9262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:37.334923 9262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:37.334968 9262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:37.334990 9262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:37.341037 9262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42145
I20260927 11:18:37.341104 9392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42145 every 8 connection(s)
I20260927 11:18:37.341434 9262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:18:37.344570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 9262
I20260927 11:18:37.344854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43351
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46403
--webserver_port=42327
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 11:18:37.349288 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:37.342123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49497 (local address 127.0.98.62:46403)
0927 11:18:37.342191 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:37.342193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:37.342368 (+ 175us) server_negotiation.cc:407] Connection header received
0927 11:18:37.342840 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:37.342841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:37.342854 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:37.342907 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:37.343786 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.344262 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.345800 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.345984 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.346041 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.349231 (+ 3190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49497: BlockingRecv error: recv got EOF from 127.0.98.2:49497 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:18:37.349351 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:37.342111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:49497)
0927 11:18:37.342264 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:37.342299 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:37.342753 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:37.342958 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:37.342961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:37.343517 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:37.343708 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.343713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.344336 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.344337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.345711 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.345714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.346079 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.346080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.346234 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.346795 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:37.346811 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:37.349156 (+ 2345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
W20260927 11:18:37.349558 9393 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46403 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:46403: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:37.354079 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:37.349808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:33361)
0927 11:18:37.349850 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:37.349859 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:37.349913 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:37.350028 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:37.350029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:37.350157 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:37.350195 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.350196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.350821 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.350822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.351318 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.351320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.351672 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.351673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.351736 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.352165 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:37.352170 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:37.354000 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 11:18:37.354139 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:37.349868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33361 (local address 127.0.98.62:46403)
0927 11:18:37.349928 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:37.349930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:37.349938 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:37.349962 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:37.349963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:37.349972 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:37.350027 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:37.350268 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.350729 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.351398 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.351591 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.351672 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.354075 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33361: BlockingRecv error: recv got EOF from 127.0.98.2:33361 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:37.358816 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:37.354404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:54533)
0927 11:18:37.354536 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:37.354557 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:37.354615 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:37.354746 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:37.354747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:37.354903 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:37.354943 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.354944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.355509 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.355510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.356312 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.356313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.356615 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.356616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.356668 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.357022 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:37.357027 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:37.358748 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
W20260927 11:18:37.358891 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:37.354479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54533 (local address 127.0.98.62:46403)
0927 11:18:37.354531 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:37.354532 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:37.354599 (+ 67us) server_negotiation.cc:407] Connection header received
0927 11:18:37.354675 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:37.354675 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:37.354682 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:37.354724 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:37.355020 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.355443 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.356387 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.356552 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.356598 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.358822 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54533: BlockingRecv error: recv got EOF from 127.0.98.2:54533 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 11:18:37.359083 9393 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 11:18:37.429361 9396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:37.429581 9396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:37.429607 9396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:37.430903 9396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:37.430958 9396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:37.430987 9396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:37.432459 9396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42327
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:37.432673 9396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:37.432862 9396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:37.438026 9402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:37.438035 9401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:37.438035 9404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:37.438426 9396 server_base.cc:1034] running on GCE node
I20260927 11:18:37.438658 9396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:37.438943 9396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:37.440145 9396 hybrid_clock.cc:648] HybridClock initialized: now 1790507917440099 us; error 36 us; skew 500 ppm
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507917, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:37.442837 9396 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 11:18:37.443300 9396 webserver.cc:460] Webserver started at http://127.0.98.3:42327/ using document root <none> and password file <none>
I20260927 11:18:37.443537 9396 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:37.443620 9396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:37.444994 9396 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:37.445600 9411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:37.445806 9396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:37.445920 9396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "87b15df824214203b68bb1395f802603"
format_stamp: "Formatted at 2026-09-27 11:18:36 on dist-test-slave-lsvg"
server_key: "901001f87dc48e923b627652ea65f018"
server_key_iv: "0537d26cae126f5b287f859f52e6308d"
server_key_version: "encryptionkey@0"
I20260927 11:18:37.446295 9396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:37.454561 9396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:37.456898 9396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:37.457124 9396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:37.457376 9396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:37.457683 9396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:37.457737 9396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:37.457778 9396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:37.457811 9396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:37.463577 9396 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43351
I20260927 11:18:37.463652 9524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43351 every 8 connection(s)
I20260927 11:18:37.464020 9396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:37.470054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 9396
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 11:18:37.471616 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:37.464774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51393 (local address 127.0.98.62:46403)
0927 11:18:37.464841 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:37.464843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:37.465041 (+ 198us) server_negotiation.cc:407] Connection header received
0927 11:18:37.465591 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:37.465592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:37.465602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:37.465659 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:37.466637 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.467136 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.468228 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.468446 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.468507 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.471560 (+ 3053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51393: BlockingRecv error: recv got EOF from 127.0.98.3:51393 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:18:37.471684 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:37.464751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:51393)
0927 11:18:37.464907 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:37.464949 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:37.465495 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:37.465721 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:37.465724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:37.466307 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:37.466511 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.466517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.467264 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.467265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.468129 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.468132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.468528 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.468529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.468640 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.469241 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:37.469249 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:37.471489 (+ 2240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
W20260927 11:18:37.472062 9525 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46403 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:46403: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:37.476737 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:37.472282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:43073)
0927 11:18:37.472352 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:37.472362 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:37.472410 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:37.472555 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:37.472556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:37.472684 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:37.472723 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.472724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.473590 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.473591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.474090 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.474092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.474442 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.474443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.474506 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.474835 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:37.474840 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:37.476670 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 11:18:37.476774 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:37.472346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43073 (local address 127.0.98.62:46403)
0927 11:18:37.472387 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:37.472388 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:37.472398 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:37.472468 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:37.472469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:37.472478 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:37.472557 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:37.472799 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.473499 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.474173 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.474372 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.474444 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.476734 (+ 2290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43073: BlockingRecv error: recv got EOF from 127.0.98.3:43073 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:18:37.478003 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:37.470593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.0.98.62:46403)
0927 11:18:37.470650 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:37.470651 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:37.470690 (+ 39us) server_negotiation.cc:407] Connection header received
0927 11:18:37.470764 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:37.470765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:37.470774 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:37.470831 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:37.471408 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.472076 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.472952 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.473320 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.473383 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.474572 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:37.474576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:37.474577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:37.474588 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:37.476287 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:37.476908 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:37.476911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:37.476911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:37.477044 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:37.477321 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:37.477322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:37.477322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:37.477615 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:37.477724 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:37.477867 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 11:18:37.477949 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:37 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:37.481345 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:37.477081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:40793)
0927 11:18:37.477121 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:37.477132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:37.477165 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:37.477277 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:37.477278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:37.477410 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:37.477455 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.477457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.478294 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.478295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.478998 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:37.479000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.479293 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.479293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:37.479326 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.479747 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:37.479750 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:37.481301 (+ 1551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 11:18:37.481410 9261 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:37.477113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40793 (local address 127.0.98.62:46403)
0927 11:18:37.477148 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:37.477150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:37.477158 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:18:37.477222 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:37.477222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:37.477229 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:37.477273 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:37.477553 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.478212 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.479063 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:37.479231 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:37.479283 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:37.481374 (+ 2091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40793: BlockingRecv error: recv got EOF from 127.0.98.3:40793 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 11:18:37.481503 9525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 11:18:38 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:38 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:38.233796 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:38.224894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:49211)
0927 11:18:38.225087 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:38.225102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:38.225164 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:38.225321 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:38.225322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:38.225472 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:38.225542 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:38.225544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:38.226606 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:38.226607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:38.227146 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:38.227147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:38.227691 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:38.227691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:38.227730 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:38.231656 (+ 3926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:38.231662 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:38.233665 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":65,"threads_started":1}
W20260927 11:18:38.233821 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:38.224949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49211 (local address 127.0.98.62:46403)
0927 11:18:38.225142 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:38.225144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:38.225157 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:18:38.225218 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:38.225219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:38.225232 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:38.225312 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:38.225617 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:38.226517 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:38.227231 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:38.227618 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:38.227690 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:38.233765 (+ 6075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49211: BlockingRecv error: recv got EOF from 127.0.98.1:49211 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":124,"threads_started":1}
Sep 27 11:18:38 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:38 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:38.364681 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:38.359597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:48451)
0927 11:18:38.359744 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:38.359762 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:38.359853 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:38.360080 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:38.360081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:38.360337 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:38.360395 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:38.360398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:38.361111 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:38.361112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:38.361684 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:38.361686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:38.362018 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:38.362019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:38.362058 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:38.362583 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:38.362588 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:38.364570 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
W20260927 11:18:38.364797 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:38.359855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48451 (local address 127.0.98.62:46403)
0927 11:18:38.359919 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:38.359921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:38.359930 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:38.359954 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:38.359956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:38.359969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:38.360080 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:38.360484 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:38.360983 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:38.361765 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:38.361949 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:38.362005 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:38.364702 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48451: BlockingRecv error: recv got EOF from 127.0.98.2:48451 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 11:18:38 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:38 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:38.486367 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:38.481842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:58323)
0927 11:18:38.482010 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:38.482020 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:38.482067 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:38.482204 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:38.482205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:38.482359 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:38.482407 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:38.482408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:38.483032 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:38.483032 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:38.483559 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:38.483560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:38.483911 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:38.483912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:38.483947 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:38.484450 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:38.484454 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:38.486297 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
W20260927 11:18:38.486439 9530 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:38.481990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58323 (local address 127.0.98.62:46403)
0927 11:18:38.482040 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:38.482046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:38.482056 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:38.482126 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:38.482126 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:38.482138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:38.482184 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:38.482473 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:38.482964 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:38.483647 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:38.483845 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:38.483902 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:38.486390 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58323: BlockingRecv error: recv got EOF from 127.0.98.3:58323 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 11:18:39 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:39 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:39.239073 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:39.234395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:52569)
0927 11:18:39.234541 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:39.234552 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:39.234623 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:39.234800 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:39.234802 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:39.234982 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:39.235028 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:39.235030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:39.235683 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:39.235684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:39.236261 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:39.236263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:39.236642 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:39.236643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:39.236682 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:39.237228 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:39.237233 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:39.238931 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
W20260927 11:18:39.239157 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:39.234439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52569 (local address 127.0.98.62:46403)
0927 11:18:39.234547 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:39.234549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:39.234604 (+ 55us) server_negotiation.cc:407] Connection header received
0927 11:18:39.234691 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:39.234692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:39.234706 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:39.234782 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:39.235099 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:39.235610 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:39.236358 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:39.236562 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:39.236637 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:39.239062 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52569: BlockingRecv error: recv got EOF from 127.0.98.1:52569 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":54,"threads_started":1}
Sep 27 11:18:39 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:39 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:39.370303 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:39.365548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:58993)
0927 11:18:39.365687 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:39.365700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:39.365755 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:39.365933 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:39.365934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:39.366139 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:39.366200 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:39.366203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:39.366904 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:39.366904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:39.367414 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:39.367415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:39.367824 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:39.367824 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:39.367862 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:39.368378 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:39.368383 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:39.370246 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
W20260927 11:18:39.370492 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:39.365671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58993 (local address 127.0.98.62:46403)
0927 11:18:39.365730 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:39.365731 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:39.365763 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:18:39.365819 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:39.365820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:39.365832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:39.365907 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:39.366325 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:39.366832 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:39.367539 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:39.367754 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:39.367819 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:39.370415 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58993: BlockingRecv error: recv got EOF from 127.0.98.2:58993 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 27 11:18:39 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:39 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:39.491998 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:39.486912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:60539)
0927 11:18:39.487084 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:39.487098 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:39.487171 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:39.487347 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:39.487347 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:39.487541 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:39.487587 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:39.487588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:39.488359 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:39.488361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:39.488903 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:39.488905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:39.489442 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:39.489443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:39.489504 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:39.490062 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:39.490067 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:39.491893 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
W20260927 11:18:39.492054 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:39.487046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60539 (local address 127.0.98.62:46403)
0927 11:18:39.487115 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:39.487117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:39.487149 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:18:39.487247 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:39.487248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:39.487263 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:39.487340 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:39.487670 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:39.488269 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:39.489005 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:39.489342 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:39.489447 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:39.491989 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60539: BlockingRecv error: recv got EOF from 127.0.98.3:60539 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 11:18:40 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:40 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:40.244307 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:40.239517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:40909)
0927 11:18:40.239674 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:40.239685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:40.239793 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:40.239965 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:40.239966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:40.240121 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:40.240169 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:40.240171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:40.240909 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:40.240910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:40.241508 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:40.241510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:40.241827 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:40.241828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:40.241868 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:40.242316 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:40.242320 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:40.244232 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
W20260927 11:18:40.244376 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:40.239597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40909 (local address 127.0.98.62:46403)
0927 11:18:40.239702 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:40.239704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:40.239764 (+ 60us) server_negotiation.cc:407] Connection header received
0927 11:18:40.239858 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:40.239858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:40.239868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:40.239937 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:40.240266 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:40.240842 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:40.241582 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:40.241767 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:40.241816 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:40.244322 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40909: BlockingRecv error: recv got EOF from 127.0.98.1:40909 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":48,"threads_started":1}
Sep 27 11:18:40 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:40 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:40.375275 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:40.370795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:56335)
0927 11:18:40.370952 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:40.370966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:40.371050 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:40.371217 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:40.371219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:40.371376 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:40.371438 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:40.371440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:40.372073 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:40.372073 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:40.372599 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:40.372600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:40.372910 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:40.372911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:40.372962 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:40.373568 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:40.373574 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:40.375212 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":59,"threads_started":1}
W20260927 11:18:40.375455 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:40.370977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56335 (local address 127.0.98.62:46403)
0927 11:18:40.371050 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:40.371052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:40.371062 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:40.371110 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:40.371111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:40.371120 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:40.371171 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:40.371513 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:40.372004 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:40.372670 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:40.372848 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:40.372897 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:40.375382 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56335: BlockingRecv error: recv got EOF from 127.0.98.2:56335 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 11:18:40 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:40 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 11:18:40.497471 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:40.492673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54153 (local address 127.0.98.62:46403)
0927 11:18:40.492719 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:40.492720 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:40.492761 (+ 41us) server_negotiation.cc:407] Connection header received
0927 11:18:40.492838 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:40.492839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:40.492852 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:40.492898 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:40.493223 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:40.493710 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:40.494506 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:40.494699 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:40.494760 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:40.497415 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54153: BlockingRecv error: recv got EOF from 127.0.98.3:54153 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:18:40.497465 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:40.492562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:54153)
0927 11:18:40.492700 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:40.492717 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:40.492771 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:40.492919 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:40.492920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:40.493111 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:40.493158 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:40.493160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:40.493870 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:40.493871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:40.494409 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:40.494411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:40.494785 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:40.494786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:40.494840 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:40.495380 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:40.495405 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:40.497340 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
Sep 27 11:18:41 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:41 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:41.249557 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:41.244920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:36165)
0927 11:18:41.245086 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:41.245099 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:41.245170 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:41.245330 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:41.245331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:41.245576 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:41.245624 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:41.245627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:41.246262 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:41.246263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:41.246791 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:41.246792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:41.247116 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:41.247117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:41.247155 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:41.247621 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:41.247626 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:41.249435 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
W20260927 11:18:41.249655 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:41.244948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36165 (local address 127.0.98.62:46403)
0927 11:18:41.245104 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:41.245106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:41.245180 (+ 74us) server_negotiation.cc:407] Connection header received
0927 11:18:41.245217 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:41.245217 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:41.245232 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:41.245305 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:41.245693 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:41.246189 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:41.246870 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:41.247053 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:41.247104 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:41.249532 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36165: BlockingRecv error: recv got EOF from 127.0.98.1:36165 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":51,"threads_started":1}
Sep 27 11:18:41 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:41 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:41.380761 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:41.375866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:58573)
0927 11:18:41.376029 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:41.376040 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:41.376100 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:41.376274 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:41.376275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:41.376490 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:41.376554 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:41.376556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:41.377370 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:41.377371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:41.377971 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:41.377973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:41.378386 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:41.378387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:41.378441 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:41.378896 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:41.378900 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:41.380662 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
W20260927 11:18:41.380836 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:41.375955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58573 (local address 127.0.98.62:46403)
0927 11:18:41.376024 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:41.376026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:41.376098 (+ 72us) server_negotiation.cc:407] Connection header received
0927 11:18:41.376154 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:41.376154 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:41.376165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:41.376226 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:41.376635 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:41.377290 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:41.378065 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:41.378296 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:41.378384 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:41.380768 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58573: BlockingRecv error: recv got EOF from 127.0.98.2:58573 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 11:18:41 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:41 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:41.502562 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:41.497960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:42355)
0927 11:18:41.498091 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:41.498102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:41.498161 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:41.498317 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:41.498317 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:41.498475 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:41.498522 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:41.498524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:41.499171 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:41.499172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:41.499695 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:41.499696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:41.500048 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:41.500048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:41.500087 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:41.500627 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:41.500632 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:41.502504 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
W20260927 11:18:41.502733 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:41.498031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42355 (local address 127.0.98.62:46403)
0927 11:18:41.498127 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:41.498129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:41.498149 (+ 20us) server_negotiation.cc:407] Connection header received
0927 11:18:41.498233 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:41.498234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:41.498245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:41.498300 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:41.498588 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:41.499098 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:41.499784 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:41.499975 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:41.500036 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:41.502649 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42355: BlockingRecv error: recv got EOF from 127.0.98.3:42355 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 11:18:42 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:42 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:42.254881 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:42.250054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:49953)
0927 11:18:42.250239 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:42.250250 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:42.250334 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:42.250506 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:42.250507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:42.250681 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:42.250731 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:42.250733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:42.251393 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:42.251394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:42.251937 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:42.251939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:42.252282 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:42.252283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:42.252337 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:42.252915 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:42.252919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:42.254810 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
W20260927 11:18:42.254987 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:42.250158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49953 (local address 127.0.98.62:46403)
0927 11:18:42.250293 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:42.250294 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:42.250316 (+ 22us) server_negotiation.cc:407] Connection header received
0927 11:18:42.250391 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:42.250392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:42.250402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:42.250453 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:42.250810 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:42.251302 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:42.252011 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:42.252193 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:42.252242 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:42.254908 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49953: BlockingRecv error: recv got EOF from 127.0.98.1:49953 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
Sep 27 11:18:42 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:42 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:42.386108 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:42.381359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:37165)
0927 11:18:42.381482 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:42.381493 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:42.381558 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:42.381719 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:42.381720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:42.381886 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:42.381935 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:42.381937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:42.382553 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:42.382553 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:42.383059 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:42.383060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:42.383607 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:42.383608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:42.383675 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:42.384152 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:42.384157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:42.386050 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
W20260927 11:18:42.386228 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:42.381489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37165 (local address 127.0.98.62:46403)
0927 11:18:42.381558 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:42.381559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:42.381568 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:42.381612 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:42.381612 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:42.381622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:42.381679 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:42.382001 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:42.382487 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:42.383190 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:42.383527 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:42.383593 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:42.386148 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37165: BlockingRecv error: recv got EOF from 127.0.98.2:37165 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 11:18:42 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:42 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:42.507467 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:42.503064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:39737)
0927 11:18:42.503266 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:42.503281 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:42.503362 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:42.503482 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:42.503482 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:42.503629 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:42.503675 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:42.503677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:42.504353 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:42.504354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:42.504863 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:42.504865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:42.505216 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:42.505216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:42.505256 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:42.505769 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:42.505773 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:42.507412 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":111,"threads_started":1}
W20260927 11:18:42.507645 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:42.503229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39737 (local address 127.0.98.62:46403)
0927 11:18:42.503284 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:42.503286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:42.503341 (+ 55us) server_negotiation.cc:407] Connection header received
0927 11:18:42.503423 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:42.503423 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:42.503434 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:42.503477 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:42.503740 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:42.504275 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:42.504935 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:42.505156 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:42.505211 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:42.507533 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39737: BlockingRecv error: recv got EOF from 127.0.98.3:39737 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 27 11:18:43 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:43 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:43.263049 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:43.255416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:49609)
0927 11:18:43.255543 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:43.255553 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:43.255599 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:43.255738 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:43.255739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:43.255912 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:43.255958 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:43.255960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:43.256576 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:43.256577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:43.257112 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:43.257113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:43.257433 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:43.257434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:43.257471 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:43.261242 (+ 3771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:43.261247 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:43.262990 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
W20260927 11:18:43.263230 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:43.255466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49609 (local address 127.0.98.62:46403)
0927 11:18:43.255592 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:43.255594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:43.255603 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:43.255643 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:43.255644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:43.255654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:43.255720 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:43.256016 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:43.256510 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:43.257176 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:43.257369 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:43.257420 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:43.263153 (+ 5733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49609: BlockingRecv error: recv got EOF from 127.0.98.1:49609 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
Sep 27 11:18:43 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:43 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:43.391459 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:43.386583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:54837)
0927 11:18:43.386751 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:43.386770 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:43.386851 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:43.387003 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:43.387004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:43.387188 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:43.387232 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:43.387234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:43.387956 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:43.387957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:43.388502 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:43.388504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:43.388901 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:43.388901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:43.388979 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:43.389514 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:43.389519 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:43.391353 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
W20260927 11:18:43.391492 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:43.386667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54837 (local address 127.0.98.62:46403)
0927 11:18:43.386737 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:43.386739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:43.386831 (+ 92us) server_negotiation.cc:407] Connection header received
0927 11:18:43.386927 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:43.386928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:43.386940 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:43.386987 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:43.387323 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:43.387853 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:43.388573 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:43.388836 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:43.388889 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:43.391428 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54837: BlockingRecv error: recv got EOF from 127.0.98.2:54837 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 11:18:43 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:43 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:43.512794 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:43.507945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:43841)
0927 11:18:43.508116 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:43.508128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:43.508217 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:43.508400 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:43.508400 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:43.508575 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:43.508618 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:43.508619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:43.509366 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:43.509367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:43.509913 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:43.509914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:43.510318 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:43.510318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:43.510380 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:43.510873 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:43.510878 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:43.512677 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
W20260927 11:18:43.512844 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:43.508040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43841 (local address 127.0.98.62:46403)
0927 11:18:43.508115 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:43.508117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:43.508197 (+ 80us) server_negotiation.cc:407] Connection header received
0927 11:18:43.508298 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:43.508300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:43.508314 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:43.508394 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:43.508705 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:43.509268 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:43.509999 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:43.510192 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:43.510308 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:43.512761 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43841: BlockingRecv error: recv got EOF from 127.0.98.3:43841 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 11:18:44 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:44 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:44.268105 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:44.263533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:33305)
0927 11:18:44.263701 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:44.263712 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:44.263764 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:44.263898 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:44.263899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:44.264056 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:44.264103 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:44.264109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:44.264841 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:44.264842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:44.265383 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:44.265384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:44.265700 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:44.265701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:44.265738 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:44.266170 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:44.266174 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:44.268030 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
W20260927 11:18:44.268230 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:44.263633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33305 (local address 127.0.98.62:46403)
0927 11:18:44.263754 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:44.263755 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:44.263765 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:44.263810 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:44.263811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:44.263822 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:44.263884 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:44.264251 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:44.264766 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:44.265456 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:44.265640 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:44.265689 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:44.268124 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33305: BlockingRecv error: recv got EOF from 127.0.98.1:33305 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":60,"threads_started":1}
Sep 27 11:18:44 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:44 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:44.396554 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:44.391950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:50853)
0927 11:18:44.392140 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:44.392154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:44.392257 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:44.392431 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:44.392432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:44.392621 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:44.392675 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:44.392676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:44.393332 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:44.393333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:44.393872 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:44.393875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:44.394192 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:44.394193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:44.394232 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:44.394727 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:44.394732 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:44.396495 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
W20260927 11:18:44.396692 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:44.392053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50853 (local address 127.0.98.62:46403)
0927 11:18:44.392126 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:44.392128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:44.392236 (+ 108us) server_negotiation.cc:407] Connection header received
0927 11:18:44.392334 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:44.392334 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:44.392346 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:44.392408 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:44.392754 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:44.393263 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:44.393949 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:44.394131 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:44.394180 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:44.396613 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50853: BlockingRecv error: recv got EOF from 127.0.98.2:50853 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 11:18:44 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:44 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:44.517944 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:44.513297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:50587)
0927 11:18:44.513473 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:44.513482 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:44.513566 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:44.513742 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:44.513743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:44.513921 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:44.513970 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:44.513972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:44.514630 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:44.514631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:44.515156 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:44.515157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:44.515547 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:44.515548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:44.515612 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:44.516053 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:44.516057 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:44.517847 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
W20260927 11:18:44.518002 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:44.513352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50587 (local address 127.0.98.62:46403)
0927 11:18:44.513435 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:44.513438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:44.513546 (+ 108us) server_negotiation.cc:407] Connection header received
0927 11:18:44.513640 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:44.513641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:44.513661 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:44.513733 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:44.514050 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:44.514544 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:44.515270 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:44.515461 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:44.515548 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:44.517935 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50587: BlockingRecv error: recv got EOF from 127.0.98.3:50587 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 11:18:45 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:45 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:45.273206 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:45.268597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:54637)
0927 11:18:45.268710 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:45.268748 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:45.268793 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:45.268961 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:45.268961 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:45.269174 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:45.269222 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:45.269225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:45.269866 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:45.269867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:45.270390 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:45.270391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:45.270703 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:45.270703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:45.270746 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:45.271198 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:45.271203 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:45.273106 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
W20260927 11:18:45.273289 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:45.268700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54637 (local address 127.0.98.62:46403)
0927 11:18:45.268837 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:45.268841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:45.268854 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:18:45.268874 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:45.268875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:45.268884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:45.268943 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:45.269292 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:45.269799 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:45.270458 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:45.270645 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:45.270693 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:45.273195 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54637: BlockingRecv error: recv got EOF from 127.0.98.1:54637 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
Sep 27 11:18:45 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:45 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:45.401826 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:45.397086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:48103)
0927 11:18:45.397242 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:45.397257 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:45.397316 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:45.397445 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:45.397446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:45.397612 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:45.397659 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:45.397660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:45.398449 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:45.398449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:45.399143 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:45.399146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:45.399508 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:45.399509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:45.399562 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:45.399992 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:45.399997 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:45.401759 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
W20260927 11:18:45.401932 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:45.397177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48103 (local address 127.0.98.62:46403)
0927 11:18:45.397243 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:45.397245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:45.397322 (+ 77us) server_negotiation.cc:407] Connection header received
0927 11:18:45.397364 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:45.397365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:45.397376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:45.397432 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:45.397720 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:45.398364 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:45.399216 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:45.399425 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:45.399508 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:45.401858 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48103: BlockingRecv error: recv got EOF from 127.0.98.2:48103 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 11:18:45 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:45 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:45.523286 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:45.518584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:35087)
0927 11:18:45.518741 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:45.518753 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:45.518821 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:45.518981 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:45.518981 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:45.519168 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:45.519216 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:45.519219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:45.519927 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:45.519927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:45.520476 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:45.520478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:45.520863 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:45.520864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:45.520931 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:45.521463 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:45.521468 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:45.523185 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
W20260927 11:18:45.523330 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:45.518665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35087 (local address 127.0.98.62:46403)
0927 11:18:45.518742 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:45.518744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:45.518807 (+ 63us) server_negotiation.cc:407] Connection header received
0927 11:18:45.518879 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:45.518880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:45.518894 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:45.518975 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:45.519319 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:45.519831 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:45.520568 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:45.520773 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:45.520861 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:45.523267 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35087: BlockingRecv error: recv got EOF from 127.0.98.3:35087 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 11:18:46 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:46 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:46.278321 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:46.273707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:42221)
0927 11:18:46.273856 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:46.273866 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:46.273918 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:46.274104 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:46.274105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:46.274276 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:46.274326 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:46.274328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:46.274954 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:46.274955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:46.275472 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:46.275473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:46.275791 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:46.275792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:46.275829 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:46.276289 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:46.276294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:46.278234 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
W20260927 11:18:46.278442 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:46.273798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42221 (local address 127.0.98.62:46403)
0927 11:18:46.273969 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:46.273971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:46.273982 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:18:46.274005 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:46.274005 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:46.274017 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:46.274088 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:46.274394 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:46.274886 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:46.275541 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:46.275729 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:46.275777 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:46.278344 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42221: BlockingRecv error: recv got EOF from 127.0.98.1:42221 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
Sep 27 11:18:46 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:46 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:46.406720 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:46.402367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:46767)
0927 11:18:46.402515 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:46.402527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:46.402604 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:46.402747 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:46.402748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:46.402911 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:46.402959 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:46.402960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:46.403597 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:46.403598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:46.404108 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:46.404109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:46.404432 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:46.404432 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:46.404471 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:46.404919 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:46.404923 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:46.406660 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
W20260927 11:18:46.406867 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:46.402439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46767 (local address 127.0.98.62:46403)
0927 11:18:46.402530 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:46.402532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:46.402604 (+ 72us) server_negotiation.cc:407] Connection header received
0927 11:18:46.402649 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:46.402649 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:46.402660 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:46.402724 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:46.403039 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:46.403531 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:46.404181 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:46.404368 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:46.404418 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:46.406788 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46767: BlockingRecv error: recv got EOF from 127.0.98.2:46767 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 11:18:46 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:46 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:46.528712 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:46.523783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:51341)
0927 11:18:46.523974 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:46.523986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:46.524083 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:46.524279 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:46.524280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:46.524460 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:46.524507 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:46.524509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:46.525287 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:46.525288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:46.525829 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:46.525831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:46.526201 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:46.526202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:46.526275 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:46.526751 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:46.526756 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:46.528651 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
W20260927 11:18:46.528851 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:46.523887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51341 (local address 127.0.98.62:46403)
0927 11:18:46.523980 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:46.523982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:46.524061 (+ 79us) server_negotiation.cc:407] Connection header received
0927 11:18:46.524170 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:46.524171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:46.524186 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:46.524270 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:46.524599 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:46.525203 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:46.525921 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:46.526127 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:46.526202 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:46.528768 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51341: BlockingRecv error: recv got EOF from 127.0.98.3:51341 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 11:18:47 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:47 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:47.283422 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:47.278796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:37827)
0927 11:18:47.278935 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:47.278945 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:47.279025 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:47.279195 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:47.279196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:47.279440 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:47.279501 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:47.279503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:47.280168 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:47.280169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:47.280697 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:47.280698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:47.281008 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:47.281008 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:47.281090 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:47.281619 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:47.281624 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:47.283353 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
W20260927 11:18:47.283512 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:47.278835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37827 (local address 127.0.98.62:46403)
0927 11:18:47.278996 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:47.278998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:47.279008 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:47.279100 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:47.279100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:47.279114 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:47.279197 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:47.279589 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:47.280086 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:47.280765 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:47.280944 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:47.280995 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:47.283442 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37827: BlockingRecv error: recv got EOF from 127.0.98.1:37827 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
Sep 27 11:18:47 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:47 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:47.412058 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:47.407307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:50497)
0927 11:18:47.407487 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:47.407502 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:47.407607 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:47.407809 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:47.407810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:47.408030 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:47.408089 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:47.408092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:47.408770 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:47.408771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:47.409344 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:47.409346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:47.409656 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:47.409656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:47.409696 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:47.410201 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:47.410207 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:47.412001 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":74,"threads_started":1}
W20260927 11:18:47.412174 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:47.407424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50497 (local address 127.0.98.62:46403)
0927 11:18:47.407524 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:47.407527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:47.407586 (+ 59us) server_negotiation.cc:407] Connection header received
0927 11:18:47.407699 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:47.407700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:47.407714 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:47.407781 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:47.408190 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:47.408694 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:47.409409 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:47.409590 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:47.409641 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:47.412098 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50497: BlockingRecv error: recv got EOF from 127.0.98.2:50497 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 11:18:47 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:47 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:47.533914 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:47.529201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:44831)
0927 11:18:47.529353 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:47.529367 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:47.529428 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:47.529585 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:47.529586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:47.529783 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:47.529830 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:47.529832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:47.530504 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:47.530505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:47.531039 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:47.531040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:47.531417 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:47.531418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:47.531480 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:47.531947 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:47.531952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:47.533820 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
W20260927 11:18:47.533962 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:47.529337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44831 (local address 127.0.98.62:46403)
0927 11:18:47.529393 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:47.529395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:47.529408 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:18:47.529490 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:47.529491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:47.529505 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:47.529578 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:47.529915 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:47.530415 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:47.531129 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:47.531339 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:47.531419 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:47.533896 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44831: BlockingRecv error: recv got EOF from 127.0.98.3:44831 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 11:18:48 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:48 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:48.293854 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:48.283939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:36309)
0927 11:18:48.284061 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:48.284076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:48.284137 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:48.284276 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:48.284277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:48.284441 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:48.284488 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:48.284490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:48.285140 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:48.285141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:48.285662 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:48.285664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:48.286011 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:48.286012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:48.286050 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:48.291522 (+ 5472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:48.291528 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:48.293754 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
W20260927 11:18:48.293977 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:48.284003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36309 (local address 127.0.98.62:46403)
0927 11:18:48.284141 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:48.284143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:48.284176 (+ 33us) server_negotiation.cc:407] Connection header received
0927 11:18:48.284196 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:48.284197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:48.284207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:48.284256 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:48.284554 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:48.285079 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:48.285745 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:48.285940 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:48.285998 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:48.293888 (+ 7890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36309: BlockingRecv error: recv got EOF from 127.0.98.1:36309 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
Sep 27 11:18:48 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:48 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:48.417560 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:48.412588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:40137)
0927 11:18:48.412789 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:48.412801 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:48.412876 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:48.413030 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:48.413032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:48.413270 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:48.413315 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:48.413317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:48.414001 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:48.414002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:48.414536 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:48.414538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:48.414914 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:48.414915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:48.414974 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:48.415427 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:48.415431 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:48.417486 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":120,"threads_started":1}
W20260927 11:18:48.417677 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:48.412693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40137 (local address 127.0.98.62:46403)
0927 11:18:48.412785 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:48.412787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:48.412854 (+ 67us) server_negotiation.cc:407] Connection header received
0927 11:18:48.412935 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:48.412935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:48.412948 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:48.413022 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:48.413402 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:48.413906 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:48.414624 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:48.414826 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:48.414913 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:48.417581 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40137: BlockingRecv error: recv got EOF from 127.0.98.2:40137 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 11:18:48 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:48 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:48.538789 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:48.534426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:47475)
0927 11:18:48.534590 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:48.534607 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:48.534668 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:48.534837 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:48.534838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:48.535045 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:48.535095 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:48.535097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:48.535746 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:48.535747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:48.536284 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:48.536286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:48.536597 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:48.536597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:48.536638 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:48.537093 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:48.537098 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:48.538734 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":110,"threads_started":1}
W20260927 11:18:48.538933 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:48.534512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47475 (local address 127.0.98.62:46403)
0927 11:18:48.534561 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:48.534563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:48.534678 (+ 115us) server_negotiation.cc:407] Connection header received
0927 11:18:48.534706 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:48.534706 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:48.534717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:48.534769 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:48.535176 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:48.535669 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:48.536360 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:48.536536 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:48.536585 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:48.538848 (+ 2263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47475: BlockingRecv error: recv got EOF from 127.0.98.3:47475 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 11:18:49 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:49 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:49.298950 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:49.294322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:43045)
0927 11:18:49.294477 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:49.294488 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:49.294548 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:49.294701 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:49.294702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:49.294872 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:49.294921 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:49.294923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:49.295599 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:49.295600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:49.296149 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:49.296150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:49.296497 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:49.296498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:49.296563 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:49.297079 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:49.297084 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:49.298891 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
W20260927 11:18:49.299073 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:49.294411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43045 (local address 127.0.98.62:46403)
0927 11:18:49.294534 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:49.294536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:49.294545 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:49.294622 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:49.294623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:49.294632 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:49.294679 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:49.295014 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:49.295523 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:49.296222 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:49.296411 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:49.296464 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:49.298984 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43045: BlockingRecv error: recv got EOF from 127.0.98.1:43045 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
Sep 27 11:18:49 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:49 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:49.423185 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:49.418109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:47477)
0927 11:18:49.418423 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:49.418435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:49.418537 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:49.418710 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:49.418711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:49.418896 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:49.418942 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:49.418944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:49.419625 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:49.419626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:49.420165 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:49.420167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:49.420532 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:49.420533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:49.420591 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:49.421091 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:49.421095 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:49.423118 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":118,"threads_started":1}
W20260927 11:18:49.423316 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:49.418230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47477 (local address 127.0.98.62:46403)
0927 11:18:49.418412 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:49.418414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:49.418521 (+ 107us) server_negotiation.cc:407] Connection header received
0927 11:18:49.418598 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:49.418599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:49.418612 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:49.418703 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:49.419042 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:49.419540 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:49.420258 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:49.420455 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:49.420535 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:49.423215 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47477: BlockingRecv error: recv got EOF from 127.0.98.2:47477 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
Sep 27 11:18:49 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:49 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:49.544189 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:49.539394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:46821)
0927 11:18:49.539527 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:49.539537 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:49.539597 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:49.539783 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:49.539785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:49.539991 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:49.540056 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:49.540058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:49.540752 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:49.540753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:49.541493 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:49.541495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:49.541822 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:49.541823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:49.541863 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:49.542317 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:49.542321 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:49.544133 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
W20260927 11:18:49.544319 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:49.539459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46821 (local address 127.0.98.62:46403)
0927 11:18:49.539530 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:49.539533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:49.539614 (+ 81us) server_negotiation.cc:407] Connection header received
0927 11:18:49.539662 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:49.539663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:49.539676 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:49.539752 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:49.540124 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:49.540659 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:49.541569 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:49.541753 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:49.541807 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:49.544219 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46821: BlockingRecv error: recv got EOF from 127.0.98.3:46821 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 11:18:50 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:50 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:50.304343 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:50.299514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:41807)
0927 11:18:50.299728 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:50.299740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:50.299807 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:50.299925 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:50.299925 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:50.300094 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:50.300145 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:50.300146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:50.300847 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:50.300849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:50.301434 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:50.301437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:50.301808 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:50.301808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:50.301855 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:50.302354 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:50.302358 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:50.304263 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":131,"threads_started":1}
W20260927 11:18:50.304492 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:50.299610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41807 (local address 127.0.98.62:46403)
0927 11:18:50.299723 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:50.299730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:50.299810 (+ 80us) server_negotiation.cc:407] Connection header received
0927 11:18:50.299850 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:50.299851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:50.299861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:50.299919 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:50.300244 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:50.300742 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:50.301531 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:50.301732 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:50.301788 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:50.304397 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41807: BlockingRecv error: recv got EOF from 127.0.98.1:41807 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":51,"threads_started":1}
Sep 27 11:18:50 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:50 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:50.428143 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:50.423722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:35929)
0927 11:18:50.423872 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:50.423885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:50.423957 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:50.424103 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:50.424104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:50.424261 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:50.424307 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:50.424308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:50.424932 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:50.424933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:50.425495 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:50.425497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:50.425830 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:50.425830 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:50.425868 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:50.426317 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:50.426321 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:50.428084 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
W20260927 11:18:50.428274 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:50.423866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35929 (local address 127.0.98.62:46403)
0927 11:18:50.423935 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:50.423936 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:50.423945 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:50.424020 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:50.424020 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:50.424031 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:50.424083 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:50.424370 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:50.424859 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:50.425577 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:50.425764 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:50.425820 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:50.428171 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35929: BlockingRecv error: recv got EOF from 127.0.98.2:35929 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 11:18:50 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:50 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:50.549752 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:50.544651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:40399)
0927 11:18:50.544965 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:50.544983 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:50.545099 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:50.545292 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:50.545293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:50.545474 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:50.545521 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:50.545523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:50.546212 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:50.546213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:50.546751 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:50.546753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:50.547147 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:50.547148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:50.547213 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:50.547660 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:50.547664 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:50.549670 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":122,"threads_started":1}
W20260927 11:18:50.549821 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:50.544762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40399 (local address 127.0.98.62:46403)
0927 11:18:50.544946 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:50.544948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:50.545076 (+ 128us) server_negotiation.cc:407] Connection header received
0927 11:18:50.545194 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:50.545195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:50.545216 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:50.545287 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:50.545613 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:50.546121 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:50.546841 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:50.547034 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:50.547146 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:50.549729 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40399: BlockingRecv error: recv got EOF from 127.0.98.3:40399 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Sep 27 11:18:51 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:51 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:51.309283 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:51.304873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:42163)
0927 11:18:51.305009 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:51.305020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:51.305123 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:51.305268 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:51.305269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:51.305429 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:51.305478 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:51.305481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:51.306115 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:51.306116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:51.306634 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:51.306636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:51.306950 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:51.306950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:51.306988 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:51.307445 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:51.307450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:51.309223 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
W20260927 11:18:51.309398 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:51.304973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42163 (local address 127.0.98.62:46403)
0927 11:18:51.305108 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:51.305109 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:51.305126 (+ 17us) server_negotiation.cc:407] Connection header received
0927 11:18:51.305179 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:51.305179 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:51.305188 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:51.305250 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:51.305547 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:51.306050 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:51.306705 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:51.306889 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:51.306940 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:51.309315 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42163: BlockingRecv error: recv got EOF from 127.0.98.1:42163 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":110,"threads_started":1}
Sep 27 11:18:51 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:51 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:51.433440 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:51.428732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:35613)
0927 11:18:51.428917 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:51.428931 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:51.429006 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:51.429169 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:51.429169 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:51.429360 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:51.429405 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:51.429407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:51.430096 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:51.430097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:51.430623 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:51.430625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:51.431003 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:51.431004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:51.431075 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:51.431636 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:51.431642 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:51.433384 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
W20260927 11:18:51.433545 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:51.428847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35613 (local address 127.0.98.62:46403)
0927 11:18:51.428921 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:51.428924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:51.428984 (+ 60us) server_negotiation.cc:407] Connection header received
0927 11:18:51.429096 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:51.429097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:51.429110 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:51.429164 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:51.429489 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:51.430004 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:51.430709 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:51.430912 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:51.431000 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:51.433477 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35613: BlockingRecv error: recv got EOF from 127.0.98.2:35613 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 11:18:51 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:51 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:51.554863 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:51.550285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:39981)
0927 11:18:51.550490 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:51.550503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:51.550583 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:51.550721 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:51.550722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:51.550962 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:51.551021 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:51.551024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:51.551689 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:51.551690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:51.552199 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:51.552200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:51.552515 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:51.552516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:51.552558 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:51.552994 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:51.552998 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:51.554769 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
W20260927 11:18:51.554908 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:51.550370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39981 (local address 127.0.98.62:46403)
0927 11:18:51.550436 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:51.550438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:51.550588 (+ 150us) server_negotiation.cc:407] Connection header received
0927 11:18:51.550624 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:51.550625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:51.550641 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:51.550711 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:51.551113 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:51.551616 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:51.552278 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:51.552453 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:51.552501 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:51.554843 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39981: BlockingRecv error: recv got EOF from 127.0.98.3:39981 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 27 11:18:52 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:52 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:52.315021 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:52.309887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:60959)
0927 11:18:52.310041 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:52.310053 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:52.310114 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:52.310263 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:52.310264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:52.310431 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:52.310480 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:52.310482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.311410 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.311411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:52.312097 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:52.312099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.312419 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.312420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:52.312466 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:52.313001 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:52.313006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:52.314955 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
W20260927 11:18:52.315140 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:52.309961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60959 (local address 127.0.98.62:46403)
0927 11:18:52.310110 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:52.310112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:52.310123 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:18:52.310167 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:52.310167 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:52.310177 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:52.310244 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:52.310594 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.311328 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.312167 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.312353 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.312405 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:52.315058 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60959: BlockingRecv error: recv got EOF from 127.0.98.1:60959 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
Sep 27 11:18:52 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:18:52 dist-test-slave-lsvg krb5kdc[8527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:18:52.438341 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:52.433991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:45167)
0927 11:18:52.434125 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:52.434143 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:52.434202 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:52.434330 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:52.434331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:52.434483 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:52.434529 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:52.434531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.435141 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.435142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:52.435659 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:52.435660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.435964 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.435965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:52.436012 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:52.436451 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:52.436455 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:52.438275 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:46403: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
W20260927 11:18:52.438467 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:52.434056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45167 (local address 127.0.98.62:46403)
0927 11:18:52.434131 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:52.434133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:52.434200 (+ 67us) server_negotiation.cc:407] Connection header received
0927 11:18:52.434246 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:52.434246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:52.434255 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:52.434309 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:52.434594 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.435074 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.435726 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.435903 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.435952 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:52.438367 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45167: BlockingRecv error: recv got EOF from 127.0.98.2:45167 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 11:18:52.472298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 9110
I20260927 11:18:52.476763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 9262
I20260927 11:18:52.481041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 9396
I20260927 11:18:52.485325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 9040
2026-09-27T11:18:52Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17455 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T11:18:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:52Z Disabled control of system clock
I20260927 11:18:52.512655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37539
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 11:18:52.594195 9591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:52.594411 9591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:52.594432 9591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:52.595638 9591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:52.595681 9591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:52.595695 9591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:52.595708 9591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:52.595758 9591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:52.597196 9591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:52.597457 9591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:52.597802 9591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:18:52.602807 9591 server_base.cc:1034] running on GCE node
W20260927 11:18:52.602777 9597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:52.602813 9596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:52.602768 9599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:52.603310 9591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:52.603621 9591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:52.604806 9591 hybrid_clock.cc:648] HybridClock initialized: now 1790507932604800 us; error 40 us; skew 500 ppm
I20260927 11:18:52.606135 9591 webserver.cc:460] Webserver started at http://127.0.98.62:33495/ using document root <none> and password file <none>
I20260927 11:18:52.606392 9591 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:52.606463 9591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:52.606647 9591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:52.607720 9591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "650e8f0d1b58415b8d78bfc18456972d"
format_stamp: "Formatted at 2026-09-27 11:18:52 on dist-test-slave-lsvg"
server_key: "610e49cdaf4110fb8043f526429ee1ae"
server_key_iv: "e0a2d15f1afde8a9eb3836d79d126eb3"
server_key_version: "encryptionkey@0"
I20260927 11:18:52.608114 9591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "650e8f0d1b58415b8d78bfc18456972d"
format_stamp: "Formatted at 2026-09-27 11:18:52 on dist-test-slave-lsvg"
server_key: "610e49cdaf4110fb8043f526429ee1ae"
server_key_iv: "e0a2d15f1afde8a9eb3836d79d126eb3"
server_key_version: "encryptionkey@0"
I20260927 11:18:52.609486 9591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:18:52.610467 9605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:52.610669 9591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:18:52.610781 9591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "650e8f0d1b58415b8d78bfc18456972d"
format_stamp: "Formatted at 2026-09-27 11:18:52 on dist-test-slave-lsvg"
server_key: "610e49cdaf4110fb8043f526429ee1ae"
server_key_iv: "e0a2d15f1afde8a9eb3836d79d126eb3"
server_key_version: "encryptionkey@0"
I20260927 11:18:52.610860 9591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:52.625896 9591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:52.626260 9591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:52.626425 9591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:52.630705 9591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37539
I20260927 11:18:52.630757 9657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37539 every 8 connection(s)
I20260927 11:18:52.631138 9591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:52.631831 9658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:52.634582 9658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d: Bootstrap starting.
I20260927 11:18:52.635104 9658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:52.635329 9658 log.cc:824] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:52.635924 9658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d: No bootstrap required, opened a new log
I20260927 11:18:52.637111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 9591
I20260927 11:18:52.637208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:52.637405 392 external_mini_cluster.cc:1428] Setting key 4b2463e7856b3ad1aa69df0c68b4cb84
I20260927 11:18:52.637820 9658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650e8f0d1b58415b8d78bfc18456972d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37539 } }
I20260927 11:18:52.638039 9658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:52.638082 9658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 650e8f0d1b58415b8d78bfc18456972d, State: Initialized, Role: FOLLOWER
I20260927 11:18:52.638234 9658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650e8f0d1b58415b8d78bfc18456972d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37539 } }
I20260927 11:18:52.638299 9658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:52.638320 9658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:52.638382 9658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:52.639046 9658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650e8f0d1b58415b8d78bfc18456972d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37539 } }
I20260927 11:18:52.639132 9658 leader_election.cc:302] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 650e8f0d1b58415b8d78bfc18456972d; no voters:
I20260927 11:18:52.639304 9658 leader_election.cc:288] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:52.639515 9663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:52.639564 9658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:52.639683 9663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [term 1 LEADER]: Becoming Leader. State: Replica: 650e8f0d1b58415b8d78bfc18456972d, State: Running, Role: LEADER
I20260927 11:18:52.639770 9663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650e8f0d1b58415b8d78bfc18456972d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37539 } }
I20260927 11:18:52.640111 9663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 650e8f0d1b58415b8d78bfc18456972d. Latest consensus state: current_term: 1 leader_uuid: "650e8f0d1b58415b8d78bfc18456972d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650e8f0d1b58415b8d78bfc18456972d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37539 } } }
I20260927 11:18:52.640185 9663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:52.640271 9664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "650e8f0d1b58415b8d78bfc18456972d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650e8f0d1b58415b8d78bfc18456972d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37539 } } }
I20260927 11:18:52.640504 9664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:52.640666 9672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:52.641418 9672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:52.643076 9672 catalog_manager.cc:1347] Generated new cluster ID: 8d89d6e0e4a444e29db2f69424efdef8
I20260927 11:18:52.643148 9672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:52.643945 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:52.637981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41346 (local address 127.0.98.62:37539)
0927 11:18:52.638204 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:52.638213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:52.638246 (+ 33us) server_negotiation.cc:407] Connection header received
0927 11:18:52.638390 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:52.638397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:52.638467 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:52.638611 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:52.638861 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.640183 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.641359 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.641939 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.642073 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:52.643246 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:52.643250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:52.643250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:52.643445 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:52.643499 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:52.643514 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:52.643581 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:52.643672 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 11:18:52.643747 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260927 11:18:52.653069 9672 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:52.653755 9672 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:52.656149 9672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 650e8f0d1b58415b8d78bfc18456972d: Generated new TSK 0
I20260927 11:18:52.656320 9672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:18:52.657828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37539
--builtin_ntp_servers=127.0.98.20:36251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 11:18:52.746182 9682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:52.746410 9682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:52.746447 9682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:52.747586 9682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:52.747646 9682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:52.747689 9682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:52.749004 9682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:52.749275 9682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:52.749608 9682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:52.754997 9690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:52.755128 9682 server_base.cc:1034] running on GCE node
W20260927 11:18:52.755342 9687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:52.755342 9688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:52.755774 9682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:52.756099 9682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:52.757290 9682 hybrid_clock.cc:648] HybridClock initialized: now 1790507932757270 us; error 37 us; skew 500 ppm
I20260927 11:18:52.758693 9682 webserver.cc:460] Webserver started at http://127.0.98.1:45371/ using document root <none> and password file <none>
I20260927 11:18:52.758961 9682 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:52.759042 9682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:52.759205 9682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:52.760350 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "ad41c6a8ac35486d87224832bf203375"
format_stamp: "Formatted at 2026-09-27 11:18:52 on dist-test-slave-lsvg"
server_key: "b3f2448adf0a57e15811c709d0df0f27"
server_key_iv: "173132eb9f62399be15925298957b0e5"
server_key_version: "encryptionkey@0"
I20260927 11:18:52.760749 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ad41c6a8ac35486d87224832bf203375"
format_stamp: "Formatted at 2026-09-27 11:18:52 on dist-test-slave-lsvg"
server_key: "b3f2448adf0a57e15811c709d0df0f27"
server_key_iv: "173132eb9f62399be15925298957b0e5"
server_key_version: "encryptionkey@0"
I20260927 11:18:52.762269 9682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:52.763121 9696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:52.763376 9682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:52.763473 9682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "ad41c6a8ac35486d87224832bf203375"
format_stamp: "Formatted at 2026-09-27 11:18:52 on dist-test-slave-lsvg"
server_key: "b3f2448adf0a57e15811c709d0df0f27"
server_key_iv: "173132eb9f62399be15925298957b0e5"
server_key_version: "encryptionkey@0"
I20260927 11:18:52.763535 9682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:52.778470 9682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:52.778784 9682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:52.778913 9682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:52.779134 9682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:52.779460 9682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:52.779501 9682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:52.779531 9682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:52.779556 9682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:52.786422 9682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46467
I20260927 11:18:52.786458 9809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46467 every 8 connection(s)
I20260927 11:18:52.786844 9682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:18:52.792120 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:52.787648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39335 (local address 127.0.98.62:37539)
0927 11:18:52.787738 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:52.787741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:52.788028 (+ 287us) server_negotiation.cc:407] Connection header received
0927 11:18:52.788528 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:52.788529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:52.788540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:52.788601 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:52.789366 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.789849 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.790828 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.791009 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.791063 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:52.791899 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:52.791900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:52.791900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:52.791904 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:52.791914 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:52.791917 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:52.791959 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:52.792038 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 11:18:52.792060 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 11:18:52.792323 9812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:52.787650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37539 (local address 127.0.98.1:39335)
0927 11:18:52.787871 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:52.787906 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:52.788391 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:52.788681 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:52.788685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:52.788693 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:52.789277 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:52.789282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.789933 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.789933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:52.790743 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:52.790746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.791083 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.791084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:52.791192 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:52.791772 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:52.791773 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:52.791802 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:52.791825 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:52.791970 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:52.791974 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:52.791976 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:18:52.792029 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 11:18:52.792109 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":118,"threads_started":1}
I20260927 11:18:52.792830 9810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37539
I20260927 11:18:52.792943 9810 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:52.793184 9810 heartbeater.cc:511] Master 127.0.98.62:37539 requested a full tablet report, sending...
I20260927 11:18:52.793694 9622 ts_manager.cc:194] Registered new tserver with Master: ad41c6a8ac35486d87224832bf203375 (127.0.98.1:46467)
I20260927 11:18:52.794047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 9682
I20260927 11:18:52.794121 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:52.794318 392 external_mini_cluster.cc:1428] Setting key 99d86ea0f5207dcb723bed23faf5250d
I20260927 11:18:52.794420 9622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39335
I20260927 11:18:52.795239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37539
--builtin_ntp_servers=127.0.98.20:36251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 11:18:52.878171 9813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:52.878403 9813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:52.878428 9813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:52.879730 9813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:52.879771 9813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:52.879806 9813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:52.881186 9813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:52.881369 9813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:52.881625 9813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:52.886575 9819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:52.886593 9821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:52.886644 9818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:52.886783 9813 server_base.cc:1034] running on GCE node
I20260927 11:18:52.887014 9813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:52.887215 9813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:52.888348 9813 hybrid_clock.cc:648] HybridClock initialized: now 1790507932888338 us; error 28 us; skew 500 ppm
I20260927 11:18:52.889737 9813 webserver.cc:460] Webserver started at http://127.0.98.2:33321/ using document root <none> and password file <none>
I20260927 11:18:52.889990 9813 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:52.890066 9813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:52.890224 9813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:52.891286 9813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "eaebb78f05e14c0d8c7ed74688596b4b"
format_stamp: "Formatted at 2026-09-27 11:18:52 on dist-test-slave-lsvg"
server_key: "9bdc1341b800cc28a3d4a6409f664eb1"
server_key_iv: "39af7e4bdd84ab838696c229dbcfc945"
server_key_version: "encryptionkey@0"
I20260927 11:18:52.891644 9813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eaebb78f05e14c0d8c7ed74688596b4b"
format_stamp: "Formatted at 2026-09-27 11:18:52 on dist-test-slave-lsvg"
server_key: "9bdc1341b800cc28a3d4a6409f664eb1"
server_key_iv: "39af7e4bdd84ab838696c229dbcfc945"
server_key_version: "encryptionkey@0"
I20260927 11:18:52.892926 9813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:18:52.893685 9827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:52.893863 9813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:18:52.893961 9813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "eaebb78f05e14c0d8c7ed74688596b4b"
format_stamp: "Formatted at 2026-09-27 11:18:52 on dist-test-slave-lsvg"
server_key: "9bdc1341b800cc28a3d4a6409f664eb1"
server_key_iv: "39af7e4bdd84ab838696c229dbcfc945"
server_key_version: "encryptionkey@0"
I20260927 11:18:52.894059 9813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:52.916028 9813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:52.916373 9813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:52.916540 9813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:52.916779 9813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:52.917179 9813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:52.917232 9813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:52.917275 9813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:52.917302 9813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:52.923002 9813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46285
I20260927 11:18:52.923106 9940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46285 every 8 connection(s)
I20260927 11:18:52.923420 9813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:18:52.928704 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:52.924172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46585 (local address 127.0.98.62:37539)
0927 11:18:52.924221 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:52.924223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:52.924446 (+ 223us) server_negotiation.cc:407] Connection header received
0927 11:18:52.925010 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:52.925012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:52.925025 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:52.925096 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:52.925785 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.926311 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.927320 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.927524 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.927610 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:52.928405 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:52.928406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:52.928407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:52.928412 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:52.928426 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:52.928430 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:52.928483 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:52.928596 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 11:18:52.928624 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:18:52.928922 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:52.924144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37539 (local address 127.0.98.2:46585)
0927 11:18:52.924314 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:52.924347 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:52.924889 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:52.925167 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:52.925171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:52.925181 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:52.925691 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:52.925696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.926408 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.926409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:52.927228 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:52.927230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:52.927621 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:52.927621 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:52.927761 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:52.928269 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:52.928270 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:52.928297 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:52.928320 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:52.928508 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:52.928512 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:52.928517 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:18:52.928598 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 11:18:52.928698 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260927 11:18:52.929466 9941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37539
I20260927 11:18:52.929593 9941 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:52.929819 9941 heartbeater.cc:511] Master 127.0.98.62:37539 requested a full tablet report, sending...
I20260927 11:18:52.930223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 9813
I20260927 11:18:52.930261 9622 ts_manager.cc:194] Registered new tserver with Master: eaebb78f05e14c0d8c7ed74688596b4b (127.0.98.2:46285)
I20260927 11:18:52.930313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:52.930501 392 external_mini_cluster.cc:1428] Setting key b1f6396b922ae60289fe8c6ab54c649b
I20260927 11:18:52.930691 9622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46585
I20260927 11:18:52.931387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37539
--builtin_ntp_servers=127.0.98.20:36251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 11:18:53.015679 9944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:53.015890 9944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:53.015911 9944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:53.017031 9944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:53.017131 9944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:53.017160 9944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:53.018508 9944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:53.018708 9944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:53.018966 9944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:53.024106 9952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:53.024223 9950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:53.024106 9949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:53.024382 9944 server_base.cc:1034] running on GCE node
I20260927 11:18:53.024645 9944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:53.024900 9944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:53.026048 9944 hybrid_clock.cc:648] HybridClock initialized: now 1790507933026039 us; error 43 us; skew 500 ppm
I20260927 11:18:53.027364 9944 webserver.cc:460] Webserver started at http://127.0.98.3:39407/ using document root <none> and password file <none>
I20260927 11:18:53.027643 9944 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:53.027693 9944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:53.027855 9944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:53.028965 9944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "1b6820d082984b8fa50f73fbeb628cd9"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "65ad7835a012c6a971ee478ba408f4cc"
server_key_iv: "b1068649b7bf1dc208baf72e6b92c96c"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.029419 9944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1b6820d082984b8fa50f73fbeb628cd9"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "65ad7835a012c6a971ee478ba408f4cc"
server_key_iv: "b1068649b7bf1dc208baf72e6b92c96c"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.030761 9944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:53.031453 9958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:53.031625 9944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:53.031716 9944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "1b6820d082984b8fa50f73fbeb628cd9"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "65ad7835a012c6a971ee478ba408f4cc"
server_key_iv: "b1068649b7bf1dc208baf72e6b92c96c"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.031800 9944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:53.054365 9944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:53.054733 9944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:53.054908 9944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:53.055174 9944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:53.055534 9944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:53.055591 9944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:53.055629 9944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:53.055657 9944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:53.061537 9944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33957
I20260927 11:18:53.061573 10071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33957 every 8 connection(s)
I20260927 11:18:53.061931 9944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:53.066160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 9944
I20260927 11:18:53.066282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:53.066454 392 external_mini_cluster.cc:1428] Setting key 4f87521f8a38ec835bc46da18e22dee6
I20260927 11:18:53.067356 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:53.062743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37713 (local address 127.0.98.62:37539)
0927 11:18:53.062808 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:53.062811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:53.063083 (+ 272us) server_negotiation.cc:407] Connection header received
0927 11:18:53.063569 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:53.063570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:53.063580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:53.063630 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:53.064295 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:53.064796 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:53.065809 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:53.066003 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:53.066102 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:53.067096 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:53.067097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:53.067098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:53.067103 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:53.067119 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:53.067123 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:53.067170 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:53.067248 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 11:18:53.067278 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:18:53.067610 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:53.062740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37539 (local address 127.0.98.3:37713)
0927 11:18:53.062954 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:53.062989 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:53.063480 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:53.063693 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:53.063696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:53.063704 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:53.064219 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:53.064224 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:53.064875 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:53.064876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:53.065737 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:53.065740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:53.066114 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:53.066115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:53.066255 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:53.066945 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:53.066946 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:53.066988 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:53.067018 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:53.067169 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:53.067174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:53.067177 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:53.067249 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 11:18:53.067350 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260927 11:18:53.068061 10072 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37539
I20260927 11:18:53.068157 10072 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:53.068357 10072 heartbeater.cc:511] Master 127.0.98.62:37539 requested a full tablet report, sending...
I20260927 11:18:53.068753 9622 ts_manager.cc:194] Registered new tserver with Master: 1b6820d082984b8fa50f73fbeb628cd9 (127.0.98.3:33957)
I20260927 11:18:53.069203 9622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37713
I20260927 11:18:53.078300 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 11:18:53.080339 9662 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:53.079757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.0.98.62:37539)
0927 11:18:53.079842 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:53.079845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:53.079960 (+ 115us) server_negotiation.cc:407] Connection header received
0927 11:18:53.080058 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:53.080059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:53.080073 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:53.080121 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:53.080262 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41350: BlockingRecv error: recv got EOF from 127.0.0.1:41350 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:18:53.080427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 9682
I20260927 11:18:53.085037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 9813
I20260927 11:18:53.089146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 9944
I20260927 11:18:53.093382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 9591
2026-09-27T11:18:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (606 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T11:18:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:53Z Disabled control of system clock
I20260927 11:18:53.120083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34383
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 11:18:53.200713 10083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:53.200929 10083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:53.200950 10083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:53.202147 10083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:53.202193 10083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:53.202206 10083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:53.202219 10083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:53.202231 10083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:53.203601 10083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:53.203858 10083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:53.204156 10083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:53.208902 10089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:53.209023 10088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:53.209023 10091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:53.209236 10083 server_base.cc:1034] running on GCE node
I20260927 11:18:53.209527 10083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:53.209769 10083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:53.210932 10083 hybrid_clock.cc:648] HybridClock initialized: now 1790507933210918 us; error 28 us; skew 500 ppm
I20260927 11:18:53.212188 10083 webserver.cc:460] Webserver started at http://127.0.98.62:34993/ using document root <none> and password file <none>
I20260927 11:18:53.212379 10083 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:53.212420 10083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:53.212538 10083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:53.213728 10083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "2a1dca7b730c4b2c83d28262dceb7060"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "9bcf1f76e525fdd9653653e3f9744f41"
server_key_iv: "379520539050768f1cd20b46db52ecc0"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.214097 10083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a1dca7b730c4b2c83d28262dceb7060"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "9bcf1f76e525fdd9653653e3f9744f41"
server_key_iv: "379520539050768f1cd20b46db52ecc0"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.215349 10083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:18:53.216025 10097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:53.216217 10083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:53.216306 10083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "2a1dca7b730c4b2c83d28262dceb7060"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "9bcf1f76e525fdd9653653e3f9744f41"
server_key_iv: "379520539050768f1cd20b46db52ecc0"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.216378 10083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:53.219766 10083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:53.220028 10083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:53.220166 10083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:53.224057 10083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34383
I20260927 11:18:53.224071 10149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34383 every 8 connection(s)
I20260927 11:18:53.224445 10083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:53.225123 10150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:53.227836 10150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060: Bootstrap starting.
I20260927 11:18:53.228425 10150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:53.228730 10150 log.cc:824] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:53.229455 10150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060: No bootstrap required, opened a new log
I20260927 11:18:53.230996 10150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1dca7b730c4b2c83d28262dceb7060" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34383 } }
I20260927 11:18:53.231137 10150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:53.231194 10150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1dca7b730c4b2c83d28262dceb7060, State: Initialized, Role: FOLLOWER
I20260927 11:18:53.231338 10150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1dca7b730c4b2c83d28262dceb7060" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34383 } }
I20260927 11:18:53.231397 10150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:53.231418 10150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:53.231472 10150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:53.231990 10150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1dca7b730c4b2c83d28262dceb7060" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34383 } }
I20260927 11:18:53.232103 10150 leader_election.cc:302] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a1dca7b730c4b2c83d28262dceb7060; no voters:
I20260927 11:18:53.232285 10150 leader_election.cc:288] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:53.232328 10153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:53.232589 10150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:53.232632 10153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [term 1 LEADER]: Becoming Leader. State: Replica: 2a1dca7b730c4b2c83d28262dceb7060, State: Running, Role: LEADER
I20260927 11:18:53.232724 10153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1dca7b730c4b2c83d28262dceb7060" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34383 } }
I20260927 11:18:53.233275 10154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a1dca7b730c4b2c83d28262dceb7060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1dca7b730c4b2c83d28262dceb7060" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34383 } } }
I20260927 11:18:53.233361 10154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:53.233489 10155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a1dca7b730c4b2c83d28262dceb7060. Latest consensus state: current_term: 1 leader_uuid: "2a1dca7b730c4b2c83d28262dceb7060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1dca7b730c4b2c83d28262dceb7060" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34383 } } }
I20260927 11:18:53.233582 10155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060 [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:53.233625 10158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:53.234148 10158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:53.234350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 10083
I20260927 11:18:53.234445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:53.234644 392 external_mini_cluster.cc:1428] Setting key b1e5355ccf0fd7f34f1c79c9d35e656b
I20260927 11:18:53.236269 10158 catalog_manager.cc:1347] Generated new cluster ID: eef6af66f9764c79a803a2f845eac9c3
I20260927 11:18:53.236318 10158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:18:53.238225 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:53.235407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.0.98.62:34383)
0927 11:18:53.235753 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:53.235761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:53.235789 (+ 28us) server_negotiation.cc:407] Connection header received
0927 11:18:53.235899 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:53.235906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:53.235948 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:53.236100 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:53.237677 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:53.237680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:53.237681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:53.237685 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:53.237733 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:53.237750 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:53.237821 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:53.237889 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 11:18:53.238053 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":75,"threads_started":1}
I20260927 11:18:53.244387 10158 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:53.244537 10158 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:53.257072 10158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a1dca7b730c4b2c83d28262dceb7060: Generated new TSK 0
I20260927 11:18:53.257308 10158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:18:53.262903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34383
--builtin_ntp_servers=127.0.98.20:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 11:18:53.352981 10174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:53.353231 10174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:53.353255 10174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:53.354467 10174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:53.354511 10174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:53.354540 10174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:53.355908 10174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:53.356175 10174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:53.356508 10174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:53.362303 10179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:53.362299 10182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:53.362303 10180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:53.362584 10174 server_base.cc:1034] running on GCE node
I20260927 11:18:53.362787 10174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:53.363051 10174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:53.364226 10174 hybrid_clock.cc:648] HybridClock initialized: now 1790507933364205 us; error 34 us; skew 500 ppm
I20260927 11:18:53.368950 10174 webserver.cc:460] Webserver started at http://127.0.98.1:42047/ using document root <none> and password file <none>
I20260927 11:18:53.369257 10174 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:53.369319 10174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:53.369447 10174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:53.370808 10174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "79271323901e426e92b519002d2be3bf"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "7e31f4fdfa1b412e54c46292d70a3f6a"
server_key_iv: "cdc679ed2f91afa1d83724f1eb147d62"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.371176 10174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "79271323901e426e92b519002d2be3bf"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "7e31f4fdfa1b412e54c46292d70a3f6a"
server_key_iv: "cdc679ed2f91afa1d83724f1eb147d62"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.372757 10174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:53.373639 10188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:53.373831 10174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:18:53.373919 10174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "79271323901e426e92b519002d2be3bf"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "7e31f4fdfa1b412e54c46292d70a3f6a"
server_key_iv: "cdc679ed2f91afa1d83724f1eb147d62"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.373986 10174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:53.378818 10174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:53.379046 10174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:53.379142 10174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:53.379299 10174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:53.379540 10174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:53.379570 10174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:53.379590 10174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:53.379603 10174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:53.385524 10174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34557
I20260927 11:18:53.385565 10301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34557 every 8 connection(s)
I20260927 11:18:53.385887 10174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:18:53.388856 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:53.386597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37667 (local address 127.0.98.62:34383)
0927 11:18:53.386653 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:53.386655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:53.386888 (+ 233us) server_negotiation.cc:407] Connection header received
0927 11:18:53.387351 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:53.387352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:53.387357 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:53.387424 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:53.388660 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:53.388661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:53.388662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:53.388665 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:53.388678 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:53.388682 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:53.388730 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:53.388789 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 11:18:53.388802 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:18:53.389093 10304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:53.386549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34383 (local address 127.0.98.1:37667)
0927 11:18:53.386749 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:53.386792 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:53.387265 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:53.387480 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:53.387483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:53.387492 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:53.388503 (+ 1011us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:53.388504 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:53.388542 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:53.388572 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:53.388729 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:53.388731 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:53.388734 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:53.388785 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 11:18:53.388834 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260927 11:18:53.389611 10302 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34383
I20260927 11:18:53.389698 10302 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:53.389820 10302 heartbeater.cc:511] Master 127.0.98.62:34383 requested a full tablet report, sending...
I20260927 11:18:53.390300 10114 ts_manager.cc:194] Registered new tserver with Master: 79271323901e426e92b519002d2be3bf (127.0.98.1:34557)
I20260927 11:18:53.391976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 10174
I20260927 11:18:53.392053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:53.392199 392 external_mini_cluster.cc:1428] Setting key 541bded7d0316b047eee48b8fd201540
I20260927 11:18:53.393098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34383
--builtin_ntp_servers=127.0.98.20:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 11:18:53.474958 10305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:53.475173 10305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:53.475198 10305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:53.476398 10305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:53.476439 10305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:53.476493 10305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:53.478042 10305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:53.478281 10305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:53.478570 10305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:53.483757 10311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:53.483887 10305 server_base.cc:1034] running on GCE node
W20260927 11:18:53.483865 10310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:53.483765 10313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:53.484233 10305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:53.484495 10305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:53.485642 10305 hybrid_clock.cc:648] HybridClock initialized: now 1790507933485631 us; error 30 us; skew 500 ppm
I20260927 11:18:53.486892 10305 webserver.cc:460] Webserver started at http://127.0.98.2:38897/ using document root <none> and password file <none>
I20260927 11:18:53.487136 10305 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:53.487182 10305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:53.487332 10305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:53.488397 10305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "c94d8b430cb840f6a22dbd6f8ab4d727"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "a5c26c2abeba1d02b7b58209783d8770"
server_key_iv: "182b1250b5c1abd69f0972a859d2861b"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.488754 10305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c94d8b430cb840f6a22dbd6f8ab4d727"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "a5c26c2abeba1d02b7b58209783d8770"
server_key_iv: "182b1250b5c1abd69f0972a859d2861b"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.490113 10305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:53.490819 10319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:53.491003 10305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:53.491093 10305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "c94d8b430cb840f6a22dbd6f8ab4d727"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "a5c26c2abeba1d02b7b58209783d8770"
server_key_iv: "182b1250b5c1abd69f0972a859d2861b"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.491169 10305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:53.494599 10305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:53.494865 10305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:53.495002 10305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:53.495203 10305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:53.495496 10305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:53.495527 10305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:53.495580 10305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:53.495608 10305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:53.501368 10305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40959
I20260927 11:18:53.501401 10432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40959 every 8 connection(s)
I20260927 11:18:53.501729 10305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:18:53.504580 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:53.502500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60161 (local address 127.0.98.62:34383)
0927 11:18:53.502554 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:53.502556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:53.502800 (+ 244us) server_negotiation.cc:407] Connection header received
0927 11:18:53.503230 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:53.503231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:53.503236 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:53.503296 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:53.504391 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:53.504392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:53.504392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:53.504395 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:53.504409 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:53.504412 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:53.504460 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:53.504515 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 11:18:53.504526 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:18:53.504796 10435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:53.502470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34383 (local address 127.0.98.2:60161)
0927 11:18:53.502654 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:53.502694 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:53.503150 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:53.503350 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:53.503353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:53.503361 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:53.504243 (+ 882us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:53.504243 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:53.504277 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:53.504305 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:53.504457 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:53.504460 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:53.504462 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:18:53.504513 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 11:18:53.504558 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260927 11:18:53.505347 10433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34383
I20260927 11:18:53.505447 10433 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:53.505604 10433 heartbeater.cc:511] Master 127.0.98.62:34383 requested a full tablet report, sending...
I20260927 11:18:53.505964 10114 ts_manager.cc:194] Registered new tserver with Master: c94d8b430cb840f6a22dbd6f8ab4d727 (127.0.98.2:40959)
I20260927 11:18:53.507459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 10305
I20260927 11:18:53.507567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:53.507736 392 external_mini_cluster.cc:1428] Setting key 8fe84600949037289d9fa8235217ad5a
I20260927 11:18:53.508591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34383
--builtin_ntp_servers=127.0.98.20:35233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 11:18:53.592257 10436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:53.592502 10436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:53.592561 10436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:53.593823 10436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:53.593880 10436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:53.593924 10436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:53.595336 10436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:53.595602 10436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:53.595906 10436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:53.600943 10442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:53.600930 10441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:53.601025 10436 server_base.cc:1034] running on GCE node
W20260927 11:18:53.600934 10444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:53.601397 10436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:53.601657 10436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:53.602846 10436 hybrid_clock.cc:648] HybridClock initialized: now 1790507933602830 us; error 27 us; skew 500 ppm
I20260927 11:18:53.604111 10436 webserver.cc:460] Webserver started at http://127.0.98.3:45585/ using document root <none> and password file <none>
I20260927 11:18:53.604362 10436 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:53.604441 10436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:53.604589 10436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:53.605785 10436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "e6a8910aac904d3ba37a89cf9666f609"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "4e8170f10e64d8f5a688e1b726d695e4"
server_key_iv: "4098d919949e378d17da4fe60500d2e7"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.606156 10436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e6a8910aac904d3ba37a89cf9666f609"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "4e8170f10e64d8f5a688e1b726d695e4"
server_key_iv: "4098d919949e378d17da4fe60500d2e7"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.607447 10436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:18:53.608249 10450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:53.608412 10436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:53.608517 10436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "e6a8910aac904d3ba37a89cf9666f609"
format_stamp: "Formatted at 2026-09-27 11:18:53 on dist-test-slave-lsvg"
server_key: "4e8170f10e64d8f5a688e1b726d695e4"
server_key_iv: "4098d919949e378d17da4fe60500d2e7"
server_key_version: "encryptionkey@0"
I20260927 11:18:53.608592 10436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:53.612067 10436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:53.612354 10436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:53.612500 10436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:53.612736 10436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:53.613119 10436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:53.613161 10436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:53.613202 10436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:53.613229 10436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:53.619163 10436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44131
I20260927 11:18:53.619215 10563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44131 every 8 connection(s)
I20260927 11:18:53.619601 10436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:53.622957 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:53.620360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51099 (local address 127.0.98.62:34383)
0927 11:18:53.620402 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:53.620411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:53.620710 (+ 299us) server_negotiation.cc:407] Connection header received
0927 11:18:53.621347 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:53.621349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:53.621355 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:53.621421 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:53.622665 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:53.622667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:53.622667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:18:53.622672 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:53.622687 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:18:53.622691 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:53.622766 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:53.622868 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 11:18:53.622885 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:18:53.623085 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 10436
I20260927 11:18:53.623154 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:53.620368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34383 (local address 127.0.98.3:51099)
0927 11:18:53.620583 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:53.620618 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:53.621223 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:53.621495 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:53.621499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:53.621509 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:53.622490 (+ 981us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:18:53.622491 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:53.622520 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:18:53.622548 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:53.622762 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:53.622768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:53.622771 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:18:53.622878 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 11:18:53.622942 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
I20260927 11:18:53.623174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:53.623363 392 external_mini_cluster.cc:1428] Setting key 64ab5adb244ef2df8ca2cb9d0cfcbfce
I20260927 11:18:53.624061 10564 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34383
I20260927 11:18:53.624162 10564 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:53.624311 10564 heartbeater.cc:511] Master 127.0.98.62:34383 requested a full tablet report, sending...
I20260927 11:18:53.624697 10113 ts_manager.cc:194] Registered new tserver with Master: e6a8910aac904d3ba37a89cf9666f609 (127.0.98.3:44131)
I20260927 11:18:53.635330 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 11:18:53.637300 10172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:18:53.636712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.0.98.62:34383)
0927 11:18:53.636772 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:53.636774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:53.636890 (+ 116us) server_negotiation.cc:407] Connection header received
0927 11:18:53.636975 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:53.636976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:53.636987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:53.637054 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:53.637227 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45474: BlockingRecv error: recv got EOF from 127.0.0.1:45474 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:18:53.637347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 10174
I20260927 11:18:53.642227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 10305
I20260927 11:18:53.646576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 10436
I20260927 11:18:53.650758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 10083
2026-09-27T11:18:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (557 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:53 dist-test-slave-lsvg krb5kdc[10574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:53 dist-test-slave-lsvg krb5kdc[10574](info): set up 2 sockets
Sep 27 11:18:53 dist-test-slave-lsvg krb5kdc[10574](info): commencing operation
krb5kdc: starting...
W20260927 11:18:54.698086 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:54 dist-test-slave-lsvg krb5kdc[10574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:18:54.820619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44647
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:54.900368 10587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:54.900568 10587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:54.900589 10587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:54.901762 10587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:18:54.901803 10587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:54.901823 10587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:54.901867 10587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:18:54.901897 10587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:18:54.903326 10587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:54.903580 10587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:54.903826 10587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:54.908660 10593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:54.908679 10592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:54.908666 10595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:54.908855 10587 server_base.cc:1034] running on GCE node
I20260927 11:18:54.909189 10587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:54.909471 10587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:54.910645 10587 hybrid_clock.cc:648] HybridClock initialized: now 1790507934910643 us; error 37 us; skew 500 ppm
Sep 27 11:18:54 dist-test-slave-lsvg krb5kdc[10574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:54.913106 10587 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:18:54.913496 10587 webserver.cc:460] Webserver started at http://127.0.98.62:37965/ using document root <none> and password file <none>
I20260927 11:18:54.913676 10587 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:54.913717 10587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:54.913805 10587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:54.914827 10587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "d9ec5b2697a343d88314fb69b5c365bd"
format_stamp: "Formatted at 2026-09-27 11:18:54 on dist-test-slave-lsvg"
server_key: "56cf526285241818bb776c369071bd35"
server_key_iv: "30923917f8c03045edb2280f4e87706b"
server_key_version: "encryptionkey@0"
I20260927 11:18:54.915127 10587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "d9ec5b2697a343d88314fb69b5c365bd"
format_stamp: "Formatted at 2026-09-27 11:18:54 on dist-test-slave-lsvg"
server_key: "56cf526285241818bb776c369071bd35"
server_key_iv: "30923917f8c03045edb2280f4e87706b"
server_key_version: "encryptionkey@0"
I20260927 11:18:54.916361 10587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:18:54.917025 10602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:54.917224 10587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:54.917308 10587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "d9ec5b2697a343d88314fb69b5c365bd"
format_stamp: "Formatted at 2026-09-27 11:18:54 on dist-test-slave-lsvg"
server_key: "56cf526285241818bb776c369071bd35"
server_key_iv: "30923917f8c03045edb2280f4e87706b"
server_key_version: "encryptionkey@0"
I20260927 11:18:54.917387 10587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:54.932667 10587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:54.934589 10587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:54.934700 10587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:54.938601 10587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44647
I20260927 11:18:54.938678 10654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44647 every 8 connection(s)
I20260927 11:18:54.939019 10587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:18:54.940050 10655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:54.942348 10655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd: Bootstrap starting.
I20260927 11:18:54.942852 10655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:54.943084 10655 log.cc:824] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:54.943723 10655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd: No bootstrap required, opened a new log
I20260927 11:18:54.945246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 10587
I20260927 11:18:54.945369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:18:54.945365 10655 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ec5b2697a343d88314fb69b5c365bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44647 } }
I20260927 11:18:54.945542 10655 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:54.945566 392 external_mini_cluster.cc:1428] Setting key 7ce57848af0e3232915d461cba5b971f
I20260927 11:18:54.945590 10655 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ec5b2697a343d88314fb69b5c365bd, State: Initialized, Role: FOLLOWER
I20260927 11:18:54.945734 10655 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ec5b2697a343d88314fb69b5c365bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44647 } }
I20260927 11:18:54.945842 10655 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:18:54.945879 10655 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:18:54.945933 10655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:54.946978 10655 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ec5b2697a343d88314fb69b5c365bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44647 } }
I20260927 11:18:54.947064 10655 leader_election.cc:302] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9ec5b2697a343d88314fb69b5c365bd; no voters:
I20260927 11:18:54.947235 10655 leader_election.cc:288] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:18:54.947292 10660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:54.947436 10660 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [term 1 LEADER]: Becoming Leader. State: Replica: d9ec5b2697a343d88314fb69b5c365bd, State: Running, Role: LEADER
I20260927 11:18:54.947515 10655 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:18:54.947528 10660 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ec5b2697a343d88314fb69b5c365bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44647 } }
I20260927 11:18:54.947845 10661 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9ec5b2697a343d88314fb69b5c365bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ec5b2697a343d88314fb69b5c365bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44647 } } }
I20260927 11:18:54.947932 10661 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:54.948167 10662 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9ec5b2697a343d88314fb69b5c365bd. Latest consensus state: current_term: 1 leader_uuid: "d9ec5b2697a343d88314fb69b5c365bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ec5b2697a343d88314fb69b5c365bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44647 } } }
I20260927 11:18:54.948292 10662 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd [sys.catalog]: This master's current role is: LEADER
I20260927 11:18:54.948436 10666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:18:54.949131 10666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:18:54.950836 10666 catalog_manager.cc:1347] Generated new cluster ID: 43a0c9b61e484143910f43f57173ea35
I20260927 11:18:54.950891 10666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:18:54 dist-test-slave-lsvg krb5kdc[10574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:54.957324 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:54.946103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.0.98.62:44647)
0927 11:18:54.946296 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:54.946307 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:54.946341 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:18:54.946471 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:54.946480 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:54.946557 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:54.946734 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:54.947425 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:54.949026 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:54.950130 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:54.950778 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:54.950986 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:54.953673 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:54.953677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:54.953677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:54.953689 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:54.955683 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:54.956066 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:54.956070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:54.956070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:54.956186 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:54.956385 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:54.956386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:54.956386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:54.956703 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:54.956876 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:54.956947 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 11:18:54.957108 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260927 11:18:54.969594 10666 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:18:54.970389 10666 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:18:54.973671 10666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd: Generated new TSK 0
I20260927 11:18:54.973863 10666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:18:55.004503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44647
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:55.085953 10681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:55.086191 10681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:55.086241 10681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:55.087393 10681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:55.087462 10681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:55.087508 10681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:18:55.089030 10681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:55.089315 10681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:55.089543 10681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:55.094662 10689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:55.094702 10687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:55.094686 10686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:55.094877 10681 server_base.cc:1034] running on GCE node
I20260927 11:18:55.095173 10681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:55.095391 10681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:55.096534 10681 hybrid_clock.cc:648] HybridClock initialized: now 1790507935096518 us; error 27 us; skew 500 ppm
Sep 27 11:18:55 dist-test-slave-lsvg krb5kdc[10574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:55.099027 10681 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:18:55.099481 10681 webserver.cc:460] Webserver started at http://127.0.98.1:45363/ using document root <none> and password file <none>
I20260927 11:18:55.099706 10681 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:55.099772 10681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:55.099923 10681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:55.100966 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "4fd2d90cf8584909bbb6971a29b0d126"
format_stamp: "Formatted at 2026-09-27 11:18:55 on dist-test-slave-lsvg"
server_key: "c841ea8fb5654c6861d64a8c7f62eab9"
server_key_iv: "2ca73e125c2d36f034eb1f2d065eb8ee"
server_key_version: "encryptionkey@0"
I20260927 11:18:55.101406 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4fd2d90cf8584909bbb6971a29b0d126"
format_stamp: "Formatted at 2026-09-27 11:18:55 on dist-test-slave-lsvg"
server_key: "c841ea8fb5654c6861d64a8c7f62eab9"
server_key_iv: "2ca73e125c2d36f034eb1f2d065eb8ee"
server_key_version: "encryptionkey@0"
I20260927 11:18:55.102730 10681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:18:55.103412 10696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:55.103595 10681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:55.103698 10681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "4fd2d90cf8584909bbb6971a29b0d126"
format_stamp: "Formatted at 2026-09-27 11:18:55 on dist-test-slave-lsvg"
server_key: "c841ea8fb5654c6861d64a8c7f62eab9"
server_key_iv: "2ca73e125c2d36f034eb1f2d065eb8ee"
server_key_version: "encryptionkey@0"
I20260927 11:18:55.103771 10681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:55.116189 10681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:55.118055 10681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:55.118220 10681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:55.118461 10681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:55.118769 10681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:55.118825 10681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:55.118861 10681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:55.118888 10681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:55.124747 10681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35493
I20260927 11:18:55.124814 10809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35493 every 8 connection(s)
I20260927 11:18:55.125190 10681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:18:55.129562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 10681
I20260927 11:18:55.129654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:18:55.129865 392 external_mini_cluster.cc:1428] Setting key e26bc0a59f4f66424bfc60a65548c093
Sep 27 11:18:55 dist-test-slave-lsvg krb5kdc[10574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:55.135423 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.125912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44647 (local address 127.0.98.1:47119)
0927 11:18:55.126076 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:55.126119 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:55.126596 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:55.126910 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:55.126914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:55.127484 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:55.127670 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.127675 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.128417 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.128418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.129337 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.129339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.129839 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.129840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.129939 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.130547 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:55.130557 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:55.132135 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:55.134099 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:55.134105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:55.134108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:55.134326 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:55.134545 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:55.134545 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:55.134546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:55.134642 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:55.134939 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:55.134943 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:55.135060 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 11:18:55.135142 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 11:18:55.135261 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260927 11:18:55.135444 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.126041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47119 (local address 127.0.98.62:44647)
0927 11:18:55.126097 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:55.126099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:55.126198 (+ 99us) server_negotiation.cc:407] Connection header received
0927 11:18:55.126704 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:55.126705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:55.126762 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:55.126834 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:55.127818 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.128334 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.129427 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.129733 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.129813 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.132249 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:55.132251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:55.132251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:55.132258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:55.134017 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:55.134404 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:55.134405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:55.134406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:55.134471 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:55.134710 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:55.134711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:55.134712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:55.134872 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:55.134946 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:55.135200 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 11:18:55.135307 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 11:18:55.136050 10810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44647
I20260927 11:18:55.136164 10810 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:55.136343 10810 heartbeater.cc:511] Master 127.0.98.62:44647 requested a full tablet report, sending...
I20260927 11:18:55.136967 10619 ts_manager.cc:194] Registered new tserver with Master: 4fd2d90cf8584909bbb6971a29b0d126 (127.0.98.1:35493)
I20260927 11:18:55.137987 10619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47119
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:18:55.156387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44647
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:55.236747 10815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:55.236945 10815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:55.236967 10815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:55.238121 10815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:55.238161 10815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:55.238186 10815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:18:55.239514 10815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:55.239692 10815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:55.239861 10815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:55.244791 10823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:55.244810 10820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:55.244858 10815 server_base.cc:1034] running on GCE node
W20260927 11:18:55.244803 10821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:55.245226 10815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:55.245481 10815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:55.246644 10815 hybrid_clock.cc:648] HybridClock initialized: now 1790507935246634 us; error 32 us; skew 500 ppm
Sep 27 11:18:55 dist-test-slave-lsvg krb5kdc[10574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:55.249286 10815 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:18:55.249711 10815 webserver.cc:460] Webserver started at http://127.0.98.2:35813/ using document root <none> and password file <none>
I20260927 11:18:55.249969 10815 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:55.250036 10815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:55.250185 10815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:55.251183 10815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "0a58e2f0b16a448d8debe76ca9689bea"
format_stamp: "Formatted at 2026-09-27 11:18:55 on dist-test-slave-lsvg"
server_key: "4385b2170b896b228013d1ee9b44e09e"
server_key_iv: "9da96f08965303fcf4922a76594a50d0"
server_key_version: "encryptionkey@0"
I20260927 11:18:55.251543 10815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0a58e2f0b16a448d8debe76ca9689bea"
format_stamp: "Formatted at 2026-09-27 11:18:55 on dist-test-slave-lsvg"
server_key: "4385b2170b896b228013d1ee9b44e09e"
server_key_iv: "9da96f08965303fcf4922a76594a50d0"
server_key_version: "encryptionkey@0"
I20260927 11:18:55.252817 10815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 11:18:55.253602 10830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:55.253782 10815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:55.253877 10815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "0a58e2f0b16a448d8debe76ca9689bea"
format_stamp: "Formatted at 2026-09-27 11:18:55 on dist-test-slave-lsvg"
server_key: "4385b2170b896b228013d1ee9b44e09e"
server_key_iv: "9da96f08965303fcf4922a76594a50d0"
server_key_version: "encryptionkey@0"
I20260927 11:18:55.253960 10815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:55.279937 10815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:55.281910 10815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:55.282083 10815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:55.282333 10815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:55.282639 10815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:55.282692 10815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:55.282734 10815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:55.282763 10815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:55.288496 10815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45003
I20260927 11:18:55.288542 10943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45003 every 8 connection(s)
I20260927 11:18:55.288906 10815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:18:55.291527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 10815
I20260927 11:18:55.291637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:18:55.291844 392 external_mini_cluster.cc:1428] Setting key 69af983d21a34108aa39fbc4b16ecab4
Sep 27 11:18:55 dist-test-slave-lsvg krb5kdc[10574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:55.299522 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.289713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41283 (local address 127.0.98.62:44647)
0927 11:18:55.289765 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:55.289767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:55.289944 (+ 177us) server_negotiation.cc:407] Connection header received
0927 11:18:55.290457 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:55.290458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:55.290468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:55.290527 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:55.291507 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.292329 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.293489 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.293826 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.293903 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.296327 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:55.296333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:55.296333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:55.296340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:55.297875 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:55.298412 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:55.298413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:55.298414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:55.298490 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:55.298799 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:55.298799 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:55.298800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:55.298924 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:55.299013 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:55.299315 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 11:18:55.299412 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 11:18:55.299747 10946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.289651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44647 (local address 127.0.98.2:41283)
0927 11:18:55.289819 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:55.289862 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:55.290368 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:55.290581 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:55.290585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:55.291177 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:55.291394 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.291399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.292481 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.292483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.293406 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.293408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.293908 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.293910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.294038 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.294580 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:55.294588 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:55.296214 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:55.298003 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:55.298011 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:55.298014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:55.298306 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:55.298574 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:55.298575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:55.298575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:55.298697 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:55.299002 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:55.299009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:55.299161 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 11:18:55.299296 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 11:18:55.299498 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260927 11:18:55.300254 10944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44647
I20260927 11:18:55.300345 10944 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:55.300546 10944 heartbeater.cc:511] Master 127.0.98.62:44647 requested a full tablet report, sending...
I20260927 11:18:55.301036 10619 ts_manager.cc:194] Registered new tserver with Master: 0a58e2f0b16a448d8debe76ca9689bea (127.0.98.2:45003)
I20260927 11:18:55.301429 10619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:18:55.318290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44647
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:18:55.399328 10949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:18:55.399531 10949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:18:55.399554 10949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:18:55.400686 10949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:18:55.400727 10949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:18:55.400776 10949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:18:55.402213 10949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:18:55.402446 10949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:18:55.402719 10949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:18:55.407793 10955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:18:55.407773 10957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:55.407974 10949 server_base.cc:1034] running on GCE node
W20260927 11:18:55.407804 10954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:18:55.408254 10949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:18:55.408457 10949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:18:55.409605 10949 hybrid_clock.cc:648] HybridClock initialized: now 1790507935409589 us; error 29 us; skew 500 ppm
Sep 27 11:18:55 dist-test-slave-lsvg krb5kdc[10574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:18:55.412171 10949 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:18:55.412654 10949 webserver.cc:460] Webserver started at http://127.0.98.3:38679/ using document root <none> and password file <none>
I20260927 11:18:55.412884 10949 fs_manager.cc:359] Metadata directory not provided
I20260927 11:18:55.412953 10949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:18:55.413139 10949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:18:55.414254 10949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "3b9db0e21f3549b481e5f43499fb65b7"
format_stamp: "Formatted at 2026-09-27 11:18:55 on dist-test-slave-lsvg"
server_key: "5b4be72c8758b5f6aa846f67dfb17ed1"
server_key_iv: "93e2ea5613f87af92306f3b0edce25da"
server_key_version: "encryptionkey@0"
I20260927 11:18:55.414624 10949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3b9db0e21f3549b481e5f43499fb65b7"
format_stamp: "Formatted at 2026-09-27 11:18:55 on dist-test-slave-lsvg"
server_key: "5b4be72c8758b5f6aa846f67dfb17ed1"
server_key_iv: "93e2ea5613f87af92306f3b0edce25da"
server_key_version: "encryptionkey@0"
I20260927 11:18:55.415952 10949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:18:55.416703 10964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:55.416906 10949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:18:55.417013 10949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "3b9db0e21f3549b481e5f43499fb65b7"
format_stamp: "Formatted at 2026-09-27 11:18:55 on dist-test-slave-lsvg"
server_key: "5b4be72c8758b5f6aa846f67dfb17ed1"
server_key_iv: "93e2ea5613f87af92306f3b0edce25da"
server_key_version: "encryptionkey@0"
I20260927 11:18:55.417147 10949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:18:55.429441 10949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:18:55.431329 10949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:18:55.431497 10949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:18:55.431730 10949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:18:55.432047 10949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:18:55.432101 10949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:55.432142 10949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:18:55.432170 10949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:18:55.438159 10949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45581
I20260927 11:18:55.438200 11077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45581 every 8 connection(s)
I20260927 11:18:55.438519 10949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:18:55.443257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 10949
I20260927 11:18:55.443382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:18:55.443590 392 external_mini_cluster.cc:1428] Setting key 7161cd06ad729fdc80ae454df59b54fb
Sep 27 11:18:55 dist-test-slave-lsvg krb5kdc[10574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:18:55.448235 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.439215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48733 (local address 127.0.98.62:44647)
0927 11:18:55.439304 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:55.439306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:55.439495 (+ 189us) server_negotiation.cc:407] Connection header received
0927 11:18:55.440031 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:55.440032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:55.440047 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:55.440123 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:55.441092 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.441653 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.442673 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.442912 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.443003 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.445411 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:55.445412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:55.445412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:55.445419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:55.446832 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:55.447253 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:55.447254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:55.447254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:55.447307 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:55.447514 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:55.447515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:55.447515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:55.447643 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:55.447745 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:55.448067 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 11:18:55.448130 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 11:18:55.448504 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.439185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44647 (local address 127.0.98.3:48733)
0927 11:18:55.439363 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:55.439409 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:55.439915 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:55.440197 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:55.440201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:55.440765 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:18:55.440953 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.440958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.441758 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.441759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.442566 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.442568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.443014 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.443015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.443140 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.443775 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:18:55.443785 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:18:55.445329 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:18:55.446907 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:55.446912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:55.446914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:55.447113 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:55.447366 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:18:55.447366 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:18:55.447367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:18:55.447461 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:18:55.447780 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:18:55.447786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:18:55.447910 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 11:18:55.448040 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 11:18:55.448167 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260927 11:18:55.449008 11078 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44647
I20260927 11:18:55.449129 11078 heartbeater.cc:464] Registering TS with master...
I20260927 11:18:55.449283 11078 heartbeater.cc:511] Master 127.0.98.62:44647 requested a full tablet report, sending...
I20260927 11:18:55.449649 10619 ts_manager.cc:194] Registered new tserver with Master: 3b9db0e21f3549b481e5f43499fb65b7 (127.0.98.3:45581)
I20260927 11:18:55.450074 10619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48733
I20260927 11:18:55.455178 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:18:55.462311 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.456569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.0.98.62:44647)
0927 11:18:55.456622 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:55.456624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:55.456760 (+ 136us) server_negotiation.cc:407] Connection header received
0927 11:18:55.456804 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:55.456805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:55.456813 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:55.456863 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:18:55.457558 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.458025 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.458705 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.458881 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.458929 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.459995 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:18:55.459996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:18:55.459996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:18:55.460002 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:18:55.461439 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:55.461727 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:55.461728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:55.461728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:55.461777 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:18:55.461922 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:18:55.461922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:18:55.461923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:18:55.462046 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:18:55.462118 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:55.462194 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 11:18:55.462250 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:18:55.464032 10619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:18:55.465618 10619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:18:55.471872 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.467682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35493 (local address 127.0.0.1:55222)
0927 11:18:55.468040 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:55.468048 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:55.468082 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:55.468614 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:55.468615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:55.468617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:55.468672 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.468674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.469919 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.469920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.470895 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.470897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.471572 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.471573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.471649 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.471662 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 11:18:55.471736 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 11:18:55.471788 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":11,"thread_start_us":40,"threads_started":1}
I20260927 11:18:55.471880 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.468170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.0.98.1:35493)
0927 11:18:55.468454 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:55.468459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:55.468469 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:18:55.468505 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:55.468507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:55.468522 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:55.468603 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:55.468764 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.469829 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.470997 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.471487 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.471572 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.471633 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:55.471763 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 11:18:55.471800 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":95,"threads_started":1}
I20260927 11:18:55.472198 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.467680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45581 (local address 127.0.0.1:48416)
0927 11:18:55.467904 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:55.467915 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:55.467967 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:55.468491 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:55.468492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:55.468494 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:55.468548 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.468549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.469599 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.469600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.470964 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.470967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.471947 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.471948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.472041 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.472046 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:18:55.472092 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 11:18:55.472129 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":44,"thread_start_us":52,"threads_started":1}
I20260927 11:18:55.472520 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.467920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.0.98.3:45581)
0927 11:18:55.468122 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:55.468127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:55.468139 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:18:55.468179 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:55.468183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:55.468200 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:55.468290 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:55.468629 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.469502 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.471067 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.471780 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.471853 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.471912 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:55.472374 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 11:18:55.472415 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":133,"threads_started":1}
I20260927 11:18:55.472700 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.467956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.0.98.2:45003)
0927 11:18:55.468148 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:55.468152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:55.468161 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:18:55.468203 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:55.468204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:55.468217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:55.468309 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:55.468474 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.469662 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.471067 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.471780 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.471853 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.471907 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:55.472559 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 11:18:55.472610 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260927 11:18:55.472668 10744 tablet_service.cc:1467] Processing CreateTablet for tablet c889e36a054f40f2aa3020376ef5c90e (DEFAULT_TABLE table=test-table [id=cbb268bc72b54463b19474a5eae285ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:55.473095 10744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c889e36a054f40f2aa3020376ef5c90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:55.473822 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.467657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45003 (local address 127.0.0.1:54728)
0927 11:18:55.467771 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:55.467811 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:55.467875 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:55.468324 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:55.468327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:55.468330 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:55.468402 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.468405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.469759 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.469759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.470967 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.470968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.472228 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.472228 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.472450 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.472455 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:18:55.472529 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 11:18:55.473776 (+ 1247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":64,"thread_start_us":69,"threads_started":1}
I20260927 11:18:55.474515 10878 tablet_service.cc:1467] Processing CreateTablet for tablet c889e36a054f40f2aa3020376ef5c90e (DEFAULT_TABLE table=test-table [id=cbb268bc72b54463b19474a5eae285ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:55.474804 10878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c889e36a054f40f2aa3020376ef5c90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:55.475888 11092 tablet_bootstrap.cc:492] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126: Bootstrap starting.
I20260927 11:18:55.476431 11012 tablet_service.cc:1467] Processing CreateTablet for tablet c889e36a054f40f2aa3020376ef5c90e (DEFAULT_TABLE table=test-table [id=cbb268bc72b54463b19474a5eae285ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:18:55.476608 11092 tablet_bootstrap.cc:654] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:55.476734 11012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c889e36a054f40f2aa3020376ef5c90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:18:55.477398 11092 log.cc:824] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:55.478168 11092 tablet_bootstrap.cc:492] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126: No bootstrap required, opened a new log
I20260927 11:18:55.478229 11092 ts_tablet_manager.cc:1397] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:18:55.479696 11095 tablet_bootstrap.cc:492] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7: Bootstrap starting.
I20260927 11:18:55.479812 11092 raft_consensus.cc:348] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a58e2f0b16a448d8debe76ca9689bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45003 } } peers { permanent_uuid: "3b9db0e21f3549b481e5f43499fb65b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "4fd2d90cf8584909bbb6971a29b0d126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35493 } }
I20260927 11:18:55.480012 11092 raft_consensus.cc:374] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:55.480057 11092 raft_consensus.cc:729] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd2d90cf8584909bbb6971a29b0d126, State: Initialized, Role: FOLLOWER
I20260927 11:18:55.480182 11092 consensus_queue.cc:260] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a58e2f0b16a448d8debe76ca9689bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45003 } } peers { permanent_uuid: "3b9db0e21f3549b481e5f43499fb65b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "4fd2d90cf8584909bbb6971a29b0d126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35493 } }
I20260927 11:18:55.480386 11095 tablet_bootstrap.cc:654] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:55.480463 11092 ts_tablet_manager.cc:1428] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:18:55.480507 10810 heartbeater.cc:503] Master 127.0.98.62:44647 was elected leader, sending a full tablet report...
I20260927 11:18:55.477841 11094 tablet_bootstrap.cc:492] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea: Bootstrap starting.
I20260927 11:18:55.480988 11095 log.cc:824] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:55.481252 11094 tablet_bootstrap.cc:654] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea: Neither blocks nor log segments found. Creating new log.
I20260927 11:18:55.481688 11094 log.cc:824] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea: Log is configured to *not* fsync() on all Append() calls
I20260927 11:18:55.482342 11095 tablet_bootstrap.cc:492] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7: No bootstrap required, opened a new log
I20260927 11:18:55.482359 11094 tablet_bootstrap.cc:492] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea: No bootstrap required, opened a new log
I20260927 11:18:55.482430 11095 ts_tablet_manager.cc:1397] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 11:18:55.482420 11094 ts_tablet_manager.cc:1397] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 11:18:55.483968 11095 raft_consensus.cc:348] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a58e2f0b16a448d8debe76ca9689bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45003 } } peers { permanent_uuid: "3b9db0e21f3549b481e5f43499fb65b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "4fd2d90cf8584909bbb6971a29b0d126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35493 } }
I20260927 11:18:55.483968 11094 raft_consensus.cc:348] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a58e2f0b16a448d8debe76ca9689bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45003 } } peers { permanent_uuid: "3b9db0e21f3549b481e5f43499fb65b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "4fd2d90cf8584909bbb6971a29b0d126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35493 } }
I20260927 11:18:55.484115 11094 raft_consensus.cc:374] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:55.484118 11095 raft_consensus.cc:374] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:18:55.484155 11094 raft_consensus.cc:729] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a58e2f0b16a448d8debe76ca9689bea, State: Initialized, Role: FOLLOWER
I20260927 11:18:55.484155 11095 raft_consensus.cc:729] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9db0e21f3549b481e5f43499fb65b7, State: Initialized, Role: FOLLOWER
I20260927 11:18:55.484275 11094 consensus_queue.cc:260] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a58e2f0b16a448d8debe76ca9689bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45003 } } peers { permanent_uuid: "3b9db0e21f3549b481e5f43499fb65b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "4fd2d90cf8584909bbb6971a29b0d126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35493 } }
I20260927 11:18:55.484274 11095 consensus_queue.cc:260] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a58e2f0b16a448d8debe76ca9689bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45003 } } peers { permanent_uuid: "3b9db0e21f3549b481e5f43499fb65b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "4fd2d90cf8584909bbb6971a29b0d126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35493 } }
I20260927 11:18:55.484567 11094 ts_tablet_manager.cc:1428] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:18:55.484581 11095 ts_tablet_manager.cc:1428] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 11:18:55.484579 10944 heartbeater.cc:503] Master 127.0.98.62:44647 was elected leader, sending a full tablet report...
I20260927 11:18:55.484620 11078 heartbeater.cc:503] Master 127.0.98.62:44647 was elected leader, sending a full tablet report...
W20260927 11:18:55.539537 10945 tablet.cc:2369] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:18:55.625965 10811 tablet.cc:2369] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:55.678960 11096 raft_consensus.cc:482] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:18:55.679118 11096 raft_consensus.cc:504] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a58e2f0b16a448d8debe76ca9689bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45003 } } peers { permanent_uuid: "3b9db0e21f3549b481e5f43499fb65b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "4fd2d90cf8584909bbb6971a29b0d126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35493 } }
I20260927 11:18:55.680348 11096 leader_election.cc:288] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a58e2f0b16a448d8debe76ca9689bea (127.0.98.2:45003), 3b9db0e21f3549b481e5f43499fb65b7 (127.0.98.3:45581)
I20260927 11:18:55.683384 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.679895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51127 (local address 127.0.98.2:45003)
0927 11:18:55.679951 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:55.679953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:55.680093 (+ 140us) server_negotiation.cc:407] Connection header received
0927 11:18:55.680185 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:55.680186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:55.680199 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:55.680228 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:55.680556 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.681257 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.682500 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.683023 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.683107 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.683118 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:55.683276 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 11:18:55.683316 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 11:18:55.683403 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.679716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45003 (local address 127.0.98.1:51127)
0927 11:18:55.680048 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:55.680056 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:55.680113 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:55.680292 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:55.680292 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:55.680294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:55.680394 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.680396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.681347 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.681348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.682407 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.682409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.683105 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.683106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.683181 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.683185 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:18:55.683268 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 11:18:55.683312 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260927 11:18:55.683492 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.679912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34179 (local address 127.0.98.3:45581)
0927 11:18:55.679957 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:55.679960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:55.679971 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:18:55.680031 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:55.680032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:55.680044 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:55.680110 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:18:55.680363 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.680913 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.682401 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.683042 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.683163 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.683175 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:55.683354 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 11:18:55.683405 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:18:55.683756 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.679714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45581 (local address 127.0.98.1:34179)
0927 11:18:55.679896 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:55.679907 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:18:55.679965 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:18:55.680169 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:18:55.680170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:18:55.680172 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:18:55.680266 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.680268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.681019 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.681020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.682298 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:18:55.682301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.683166 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.683167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:18:55.683251 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.683256 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:18:55.683596 (+ 340us) client_negotiation.cc:240] Negotiation successful
0927 11:18:55.683651 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":64,"threads_started":1}
I20260927 11:18:55.683954 10898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c889e36a054f40f2aa3020376ef5c90e" candidate_uuid: "4fd2d90cf8584909bbb6971a29b0d126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a58e2f0b16a448d8debe76ca9689bea" is_pre_election: true
I20260927 11:18:55.684129 10898 raft_consensus.cc:2454] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd2d90cf8584909bbb6971a29b0d126 in term 0.
I20260927 11:18:55.684089 11032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c889e36a054f40f2aa3020376ef5c90e" candidate_uuid: "4fd2d90cf8584909bbb6971a29b0d126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9db0e21f3549b481e5f43499fb65b7" is_pre_election: true
I20260927 11:18:55.684218 11032 raft_consensus.cc:2454] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fd2d90cf8584909bbb6971a29b0d126 in term 0.
I20260927 11:18:55.684394 10700 leader_election.cc:302] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a58e2f0b16a448d8debe76ca9689bea, 4fd2d90cf8584909bbb6971a29b0d126; no voters:
I20260927 11:18:55.684525 11096 raft_consensus.cc:2781] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:18:55.684573 11096 raft_consensus.cc:482] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:18:55.684603 11096 raft_consensus.cc:3037] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:55.685680 11096 raft_consensus.cc:504] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a58e2f0b16a448d8debe76ca9689bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45003 } } peers { permanent_uuid: "3b9db0e21f3549b481e5f43499fb65b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "4fd2d90cf8584909bbb6971a29b0d126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35493 } }
I20260927 11:18:55.685859 11096 leader_election.cc:288] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [CANDIDATE]: Term 1 election: Requested vote from peers 0a58e2f0b16a448d8debe76ca9689bea (127.0.98.2:45003), 3b9db0e21f3549b481e5f43499fb65b7 (127.0.98.3:45581)
I20260927 11:18:55.686017 11032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c889e36a054f40f2aa3020376ef5c90e" candidate_uuid: "4fd2d90cf8584909bbb6971a29b0d126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9db0e21f3549b481e5f43499fb65b7"
I20260927 11:18:55.686031 10898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c889e36a054f40f2aa3020376ef5c90e" candidate_uuid: "4fd2d90cf8584909bbb6971a29b0d126" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a58e2f0b16a448d8debe76ca9689bea"
I20260927 11:18:55.686113 11032 raft_consensus.cc:3037] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:55.686120 10898 raft_consensus.cc:3037] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:18:55.686848 11032 raft_consensus.cc:2454] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd2d90cf8584909bbb6971a29b0d126 in term 1.
I20260927 11:18:55.686847 10898 raft_consensus.cc:2454] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fd2d90cf8584909bbb6971a29b0d126 in term 1.
I20260927 11:18:55.687100 10699 leader_election.cc:302] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b9db0e21f3549b481e5f43499fb65b7, 4fd2d90cf8584909bbb6971a29b0d126; no voters:
I20260927 11:18:55.687230 11096 raft_consensus.cc:2781] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:18:55.687372 11096 raft_consensus.cc:686] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [term 1 LEADER]: Becoming Leader. State: Replica: 4fd2d90cf8584909bbb6971a29b0d126, State: Running, Role: LEADER
I20260927 11:18:55.687606 11096 consensus_queue.cc:237] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a58e2f0b16a448d8debe76ca9689bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45003 } } peers { permanent_uuid: "3b9db0e21f3549b481e5f43499fb65b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } } peers { permanent_uuid: "4fd2d90cf8584909bbb6971a29b0d126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35493 } }
I20260927 11:18:55.688580 10619 catalog_manager.cc:5495] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fd2d90cf8584909bbb6971a29b0d126 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4fd2d90cf8584909bbb6971a29b0d126" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a58e2f0b16a448d8debe76ca9689bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9db0e21f3549b481e5f43499fb65b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fd2d90cf8584909bbb6971a29b0d126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35493 } health_report { overall_health: HEALTHY } } }
W20260927 11:18:55.688972 11079 tablet.cc:2369] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:18:55.697942 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.695095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.0.98.1:35493)
0927 11:18:55.695138 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:18:55.695140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:18:55.695151 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:18:55.695208 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:18:55.695209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:18:55.695227 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:18:55.695283 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:18:55.695480 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.696163 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.696971 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:18:55.697282 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:18:55.697355 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:18:55.697568 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:18:55.697703 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:18:55.697802 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 11:18:55.697824 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 11:18:55.697850 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:18:55.700049 10898 raft_consensus.cc:1261] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea [term 1 FOLLOWER]: Refusing update from remote peer 4fd2d90cf8584909bbb6971a29b0d126: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:55.700049 11032 raft_consensus.cc:1261] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7 [term 1 FOLLOWER]: Refusing update from remote peer 4fd2d90cf8584909bbb6971a29b0d126: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:18:55.700367 11103 consensus_queue.cc:1035] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9db0e21f3549b481e5f43499fb65b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:55.700495 11096 consensus_queue.cc:1035] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a58e2f0b16a448d8debe76ca9689bea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:18:55.703156 11108 mvcc.cc:205] Tried to move back new op lower bound from 7333920504624746496 to 7333920504577298432. Current Snapshot: MvccSnapshot[applied={T|T < 7333920504624746496}]
I20260927 11:18:55.703809 11107 mvcc.cc:205] Tried to move back new op lower bound from 7333920504624746496 to 7333920504577298432. Current Snapshot: MvccSnapshot[applied={T|T < 7333920504624746496}]
I20260927 11:18:55.705147 10618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:55.705262 10618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:18:55.706087 10618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9ec5b2697a343d88314fb69b5c365bd: Sending DeleteTablet for 3 replicas of tablet c889e36a054f40f2aa3020376ef5c90e
I20260927 11:18:55.706422 10744 tablet_service.cc:1514] Processing DeleteTablet for tablet c889e36a054f40f2aa3020376ef5c90e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55222
I20260927 11:18:55.706418 10878 tablet_service.cc:1514] Processing DeleteTablet for tablet c889e36a054f40f2aa3020376ef5c90e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54728
I20260927 11:18:55.706638 11116 tablet_replica.cc:330] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea: stopping tablet replica
I20260927 11:18:55.706638 11115 tablet_replica.cc:330] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126: stopping tablet replica
I20260927 11:18:55.706729 11116 raft_consensus.cc:2229] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:55.706730 11115 raft_consensus.cc:2229] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:18:55.706826 11116 raft_consensus.cc:2258] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:55.706825 11115 raft_consensus.cc:2258] T c889e36a054f40f2aa3020376ef5c90e P 4fd2d90cf8584909bbb6971a29b0d126 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:55.706936 11012 tablet_service.cc:1514] Processing DeleteTablet for tablet c889e36a054f40f2aa3020376ef5c90e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:18:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48416
I20260927 11:18:55.707276 11117 tablet_replica.cc:330] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7: stopping tablet replica
I20260927 11:18:55.707429 11117 raft_consensus.cc:2229] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:18:55.707414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 10681
I20260927 11:18:55.707579 11117 raft_consensus.cc:2258] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:18:55.708364 11117 ts_tablet_manager.cc:1905] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:55.710353 11117 ts_tablet_manager.cc:1918] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:18:55.710433 11117 log.cc:1196] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/c889e36a054f40f2aa3020376ef5c90e
I20260927 11:18:55.710733 11117 ts_tablet_manager.cc:1939] T c889e36a054f40f2aa3020376ef5c90e P 3b9db0e21f3549b481e5f43499fb65b7: Deleting consensus metadata
I20260927 11:18:55.711189 10605 catalog_manager.cc:4841] TS 3b9db0e21f3549b481e5f43499fb65b7 (127.0.98.3:45581): tablet c889e36a054f40f2aa3020376ef5c90e (table test-table [id=cbb268bc72b54463b19474a5eae285ea]) successfully deleted
W20260927 11:18:55.713711 10603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35493 (error 108)
I20260927 11:18:55.713999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 10815
I20260927 11:18:55.714293 11116 ts_tablet_manager.cc:1905] T c889e36a054f40f2aa3020376ef5c90e P 0a58e2f0b16a448d8debe76ca9689bea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:18:55.717554 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.717417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35493 (local address 127.0.0.1:55244)
0927 11:18:55.717471 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:55.717525 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 11:18:55.717701 10603 catalog_manager.cc:4584] TS 4fd2d90cf8584909bbb6971a29b0d126 (127.0.98.1:35493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c889e36a054f40f2aa3020376ef5c90e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35493: connect: Connection refused (error 111)
I20260927 11:18:55.721449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 10949
I20260927 11:18:55.721514 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:18:55.721389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45003 (local address 127.0.0.1:54730)
0927 11:18:55.721440 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 11:18:55.721483 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 11:18:55.726440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 10587
2026-09-27T11:18:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2076 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:18:55 dist-test-slave-lsvg krb5kdc[11121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:18:55 dist-test-slave-lsvg krb5kdc[11121](info): set up 2 sockets
Sep 27 11:18:55 dist-test-slave-lsvg krb5kdc[11121](info): commencing operation
krb5kdc: starting...
W20260927 11:18:56.758203 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:18:56 dist-test-slave-lsvg krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:18:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:18:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 11:18:56.927326 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskuJOgQn/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/postgres -l logfile start
2026-09-27 11:18:59.500 UTC [11151] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 11:18:59.500 UTC [11151] LOG: listening on IPv4 address "127.0.98.20", port 48933
2026-09-27 11:18:59.504 UTC [11151] LOG: listening on Unix socket "/tmp/.s.PGSQL.48933"
2026-09-27 11:18:59.539 UTC [11154] LOG: database system was shut down at 2026-09-27 11:18:57 UTC
2026-09-27 11:18:59.543 UTC [11151] LOG: database system is ready to accept connections
I20260927 11:19:00.517939 392 mini_postgres.cc:96] Postgres bound to 48933
2026-09-27 11:19:00.523 UTC [11162] FATAL: database "slave" does not exist
127.0.98.20:48933 - accepting connections
I20260927 11:19:00.524506 392 mini_ranger.cc:161] Starting Ranger...
I20260927 11:19:00.539143 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 11:19:01.026809 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 11:19:01.026891 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-admin
2026-09-27 11:19:01,325 [I] DB FLAVOR :POSTGRES
2026-09-27 11:19:01,326 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 11:19:01,326 [I] Checking connection..
2026-09-27 11:19:01,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 11:19:01,572 [I] Checking connection passed.
2026-09-27 11:19:01,572 [I] --------- Verifying version history table ---------
2026-09-27 11:19:01,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 11:19:01,834 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 11:19:01,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 11:19:02,078 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 11:19:02,078 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 11:19:02,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 11:19:02.311 UTC [11226] WARNING: there is no transaction in progress
2026-09-27 11:19:02,317 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 11:19:02,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 11:19:02,576 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 11:19:02,576 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 11:19:02,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 11:19:02,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 11:19:03,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lsvg.c.gcp-upstream.internal','N') ;"
2026-09-27 11:19:03,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 11:19:03,575 [I] Table x_portal_user does not exist in database ranger
2026-09-27 11:19:03,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 11:19:03,809 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 11:19:03,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 11:19:04,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 11:19:04,277 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 11:19:04,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 11:19:05.276 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.293 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.310 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.327 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.344 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.363 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.429 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.436 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.451 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.461 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.472 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.486 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.496 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.502 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.509 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:05.515 UTC [11334] WARNING: there is no transaction in progress
2026-09-27 11:19:06,157 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 11:19:06,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 11:19:06,410 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 11:19:06,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 11:19:06,639 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 11:19:06,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 11:19:06,871 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 11:19:06,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 11:19:07,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 11:19:07,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lsvg.c.gcp-upstream.internal';"
2026-09-27 11:19:07,568 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 11:19:07,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 11:19:07,817 [I] DB_PATCHES have already been applied
I20260927 11:19:07.823444 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskuJOgQn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 11:19:07.823530 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 11:19:07.826668 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:53261
Sep 27, 2026 11:19:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 11:19:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 11:19:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 11:19:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 11:19:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 11:19:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 11:19:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 11:19:08.928037 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.404s user 0.000s sys 0.009s
Sep 27, 2026 11:19:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 11:19:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 11:19:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 11:19:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 11:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 11:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 11:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 11:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 11:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 11:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 11:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 11:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 11:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 11:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 11:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 11:19:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 11:19:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 11:19:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 11:19:30.946691 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 11:19:31 dist-test-slave-lsvg krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507971, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 11:19:31.031997 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 11:19:31.040887 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 11:19:31.966531 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 11:19:31.966621 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-kms
I20260927 11:19:31.966640 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:48933
2026-09-27 11:19:32,144 [I] DB FLAVOR :POSTGRES
2026-09-27 11:19:32,145 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 11:19:32,145 [I] Checking connection
2026-09-27 11:19:32,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 11:19:32,431 [I] connection success
2026-09-27 11:19:32,431 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 11:19:32,431 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 11:19:32,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 11:19:32,712 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 11:19:32,712 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 11:19:32,713 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 11:19:32,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48933/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 11:19:32,977 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 11:19:33.430387 392 mini_ranger_kms.cc:310] Created kms service
I20260927 11:19:33.646971 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 11:19:33.691183 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 11:19:33 dist-test-slave-lsvg krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507971, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:19:34.093917 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 11:19:34.094064 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 11:19:34.094089 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 11:19:34.097497 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11514
I20260927 11:19:34.097638 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:51743
Sep 27, 2026 11:19:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 11:19:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 11:19:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 11:19:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 11:19:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 11:19:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 11:19:36.411314 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.379s user 0.008s sys 0.152s
I20260927 11:19:36.411432 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 11:19:36.411537 392 mini_ranger_kms.cc:401] 127.0.98.20:51743/kms/v1/keys
Sep 27 11:19:36 dist-test-slave-lsvg krb5kdc[11121](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790507976, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 11:19:36 dist-test-slave-lsvg krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507976, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 11:19:37 dist-test-slave-lsvg krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507976, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 11:19:39 dist-test-slave-lsvg krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:19:39.617780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34233
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34233
--ranger_config_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51743/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:19:39.699069 11565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:19:39.699306 11565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:19:39.699340 11565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:19:39.700441 11565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:19:39.700502 11565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:19:39.700525 11565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:19:39.700548 11565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:19:39.700575 11565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:19:39.702108 11565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:51743/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34233
--ranger_config_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:19:39.702370 11565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:19:39.702582 11565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:19:39.708421 11571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:19:39.708465 11573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:19:39.708496 11565 server_base.cc:1034] running on GCE node
W20260927 11:19:39.708662 11570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:19:39.708961 11565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:19:39.709260 11565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:19:39.710457 11565 hybrid_clock.cc:648] HybridClock initialized: now 1790507979710434 us; error 38 us; skew 500 ppm
Sep 27 11:19:39 dist-test-slave-lsvg krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:19:39.713610 11565 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:19:39.714118 11565 webserver.cc:460] Webserver started at http://127.0.98.62:37387/ using document root <none> and password file <none>
I20260927 11:19:39.714357 11565 fs_manager.cc:359] Metadata directory not provided
I20260927 11:19:39.714408 11565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:19:39.714531 11565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:19:39 dist-test-slave-lsvg krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:19:39.772557 11565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "1748b329a57442b78d0784298395659c"
format_stamp: "Formatted at 2026-09-27 11:19:39 on dist-test-slave-lsvg"
server_key: "2c147d84a8d0b793e5f0f94790b20633"
server_key_iv: "323e51e7eec4966636690d0426881d77"
server_key_version: "kuduclusterkey@0"
I20260927 11:19:39.772962 11565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "1748b329a57442b78d0784298395659c"
format_stamp: "Formatted at 2026-09-27 11:19:39 on dist-test-slave-lsvg"
server_key: "2c147d84a8d0b793e5f0f94790b20633"
server_key_iv: "323e51e7eec4966636690d0426881d77"
server_key_version: "kuduclusterkey@0"
I20260927 11:19:39.774402 11565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:19:39.788733 11580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:19:39.789018 11565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:19:39.789155 11565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "1748b329a57442b78d0784298395659c"
format_stamp: "Formatted at 2026-09-27 11:19:39 on dist-test-slave-lsvg"
server_key: "2c147d84a8d0b793e5f0f94790b20633"
server_key_iv: "323e51e7eec4966636690d0426881d77"
server_key_version: "kuduclusterkey@0"
I20260927 11:19:39.789266 11565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:19:39.801549 11565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:19:39.803576 11565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:19:39.803740 11565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:19:39.808866 11565 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34233
I20260927 11:19:39.808894 11632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34233 every 8 connection(s)
I20260927 11:19:39.809321 11565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:19:39.809986 11633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:19:39.812247 11633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c: Bootstrap starting.
I20260927 11:19:39.813282 11633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c: Neither blocks nor log segments found. Creating new log.
I20260927 11:19:39.813624 11633 log.cc:824] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c: Log is configured to *not* fsync() on all Append() calls
I20260927 11:19:39.813691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 11565
I20260927 11:19:39.813789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:19:39.813982 392 external_mini_cluster.cc:1428] Setting key 063e57ae82fa9db9cfdad36dba982c19
I20260927 11:19:39.814440 11633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c: No bootstrap required, opened a new log
I20260927 11:19:39.816993 11633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1748b329a57442b78d0784298395659c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34233 } }
I20260927 11:19:39.817194 11633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:19:39.817263 11633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1748b329a57442b78d0784298395659c, State: Initialized, Role: FOLLOWER
I20260927 11:19:39.817412 11633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1748b329a57442b78d0784298395659c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34233 } }
I20260927 11:19:39.817505 11633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:19:39.817564 11633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:19:39.817613 11633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:19:39.818274 11633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1748b329a57442b78d0784298395659c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34233 } }
I20260927 11:19:39.818389 11633 leader_election.cc:302] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1748b329a57442b78d0784298395659c; no voters:
I20260927 11:19:39.818586 11633 leader_election.cc:288] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:19:39.818605 11638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:19:39.818836 11638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [term 1 LEADER]: Becoming Leader. State: Replica: 1748b329a57442b78d0784298395659c, State: Running, Role: LEADER
I20260927 11:19:39.818943 11633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:19:39.818960 11638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1748b329a57442b78d0784298395659c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34233 } }
I20260927 11:19:39.819312 11638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1748b329a57442b78d0784298395659c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1748b329a57442b78d0784298395659c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34233 } } }
I20260927 11:19:39.819384 11638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [sys.catalog]: This master's current role is: LEADER
I20260927 11:19:39.819507 11638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1748b329a57442b78d0784298395659c. Latest consensus state: current_term: 1 leader_uuid: "1748b329a57442b78d0784298395659c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1748b329a57442b78d0784298395659c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34233 } } }
I20260927 11:19:39.819576 11638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c [sys.catalog]: This master's current role is: LEADER
I20260927 11:19:39.820748 11633 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 27 11:19:39 dist-test-slave-lsvg krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:19:39.824465 11642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:19:39.825500 11642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:19:39.829555 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:39.814529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.0.98.62:34233)
0927 11:19:39.814707 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 11:19:39.814716 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:19:39.814746 (+ 30us) server_negotiation.cc:407] Connection header received
0927 11:19:39.815633 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:19:39.815644 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:19:39.815707 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:19:39.815867 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:19:39.816563 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:39.817948 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:39.818976 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:39.819589 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:39.819669 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:39.823183 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:19:39.823244 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:19:39.823244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:19:39.823355 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:19:39.827243 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:19:39.827713 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:19:39.827716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:19:39.827716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:19:39.827820 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:19:39.828044 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:19:39.828045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:19:39.828045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:19:39.828444 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:19:39.828592 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 11:19:39.828856 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 11:19:39.829218 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260927 11:19:39.829768 11642 catalog_manager.cc:1347] Generated new cluster ID: 02a894ce34ad402aa54feddefeda5eb3
I20260927 11:19:39.829959 11642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:19:39.858479 11642 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:19:39.859290 11642 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:19:39.863956 11642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c: Generated new TSK 0
I20260927 11:19:39.864230 11642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:19:40.544135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51743/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34233
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:19:40.635254 11678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:19:40.635481 11678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:19:40.635519 11678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:19:40.636683 11678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:19:40.636745 11678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:19:40.636790 11678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:19:40.638351 11678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:51743/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:19:40.638612 11678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:19:40.638860 11678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:19:40.644198 11684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:19:40.644198 11686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:19:40.644500 11683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:19:40.644994 11678 server_base.cc:1034] running on GCE node
I20260927 11:19:40.645284 11678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:19:40.645578 11678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:19:40.646734 11678 hybrid_clock.cc:648] HybridClock initialized: now 1790507980646716 us; error 35 us; skew 500 ppm
Sep 27 11:19:40 dist-test-slave-lsvg krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:19:40.649637 11678 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:19:40.650242 11678 webserver.cc:460] Webserver started at http://127.0.98.1:39109/ using document root <none> and password file <none>
I20260927 11:19:40.650585 11678 fs_manager.cc:359] Metadata directory not provided
I20260927 11:19:40.650686 11678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:19:40.650851 11678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:19:40 dist-test-slave-lsvg krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:19:40.665530 11678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "378a87abb750424c9427bc6438fe4a92"
format_stamp: "Formatted at 2026-09-27 11:19:40 on dist-test-slave-lsvg"
server_key: "38015ca811dc9989b712fce4b1d4f8bb"
server_key_iv: "84ea6fdfe821a290b371943d1f6b2d8d"
server_key_version: "kuduclusterkey@0"
I20260927 11:19:40.666061 11678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "378a87abb750424c9427bc6438fe4a92"
format_stamp: "Formatted at 2026-09-27 11:19:40 on dist-test-slave-lsvg"
server_key: "38015ca811dc9989b712fce4b1d4f8bb"
server_key_iv: "84ea6fdfe821a290b371943d1f6b2d8d"
server_key_version: "kuduclusterkey@0"
I20260927 11:19:40.669437 11678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 11:19:40.681691 11693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:19:40.681903 11678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:19:40.682005 11678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "378a87abb750424c9427bc6438fe4a92"
format_stamp: "Formatted at 2026-09-27 11:19:40 on dist-test-slave-lsvg"
server_key: "38015ca811dc9989b712fce4b1d4f8bb"
server_key_iv: "84ea6fdfe821a290b371943d1f6b2d8d"
server_key_version: "kuduclusterkey@0"
I20260927 11:19:40.682113 11678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:19:40.698690 11678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:19:40.700918 11678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:19:40.701090 11678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:19:40.701361 11678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:19:40.701727 11678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:19:40.701781 11678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:19:40.701845 11678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:19:40.701889 11678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:19:40.709910 11678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39045
I20260927 11:19:40.710023 11806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39045 every 8 connection(s)
I20260927 11:19:40.710382 11678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:19:40 dist-test-slave-lsvg krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:19:40.720026 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:40.711267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33611 (local address 127.0.98.62:34233)
0927 11:19:40.711409 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 11:19:40.711411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:19:40.711607 (+ 196us) server_negotiation.cc:407] Connection header received
0927 11:19:40.712187 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:19:40.712188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:19:40.712203 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:19:40.712281 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:19:40.712846 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:40.713450 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:40.714513 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:40.714724 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:40.714806 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:40.716876 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:19:40.716877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:19:40.716877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:19:40.716884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:19:40.718449 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:19:40.718935 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:19:40.718936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:19:40.718937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:19:40.719000 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:19:40.719245 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:19:40.719246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:19:40.719247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:19:40.719416 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:19:40.719473 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:19:40.719779 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 11:19:40.719929 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260927 11:19:40.720214 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:40.711266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34233 (local address 127.0.98.1:33611)
0927 11:19:40.711473 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 11:19:40.711518 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 11:19:40.712068 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:19:40.712356 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:19:40.712360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:19:40.712549 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:19:40.712751 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:40.712755 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:40.713541 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:40.713542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:40.714416 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:40.714418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:40.714815 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:40.714816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:40.714942 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:40.715518 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:19:40.715527 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:19:40.716778 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:19:40.718542 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:19:40.718548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:19:40.718550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:19:40.718874 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:19:40.719063 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:19:40.719064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:19:40.719065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:19:40.719180 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:19:40.719504 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:19:40.719509 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:19:40.719649 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 11:19:40.719757 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 11:19:40.719965 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260927 11:19:40.720607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 11678
I20260927 11:19:40.720734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:19:40.720970 392 external_mini_cluster.cc:1428] Setting key 122b76823bf6b3a39d38d6ce9bfed291
I20260927 11:19:40.721447 11807 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34233
I20260927 11:19:40.721562 11807 heartbeater.cc:464] Registering TS with master...
I20260927 11:19:40.721766 11807 heartbeater.cc:511] Master 127.0.98.62:34233 requested a full tablet report, sending...
I20260927 11:19:40.723528 11596 ts_manager.cc:194] Registered new tserver with Master: 378a87abb750424c9427bc6438fe4a92 (127.0.98.1:39045)
I20260927 11:19:40.724368 11596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33611
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:19:40.755451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51743/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34233
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:19:40.872910 11815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:19:40.873240 11815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:19:40.873279 11815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:19:40.875052 11815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:19:40.875115 11815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:19:40.875154 11815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:19:40.877408 11815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:51743/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:19:40.877650 11815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:19:40.877946 11815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:19:40.884095 11825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:19:40.884163 11823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:19:40.884509 11822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:19:40.884734 11815 server_base.cc:1034] running on GCE node
I20260927 11:19:40.884936 11815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:19:40.885260 11815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:19:40.886436 11815 hybrid_clock.cc:648] HybridClock initialized: now 1790507980886419 us; error 34 us; skew 500 ppm
Sep 27 11:19:40 dist-test-slave-lsvg krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:19:40.889742 11815 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:19:40.890393 11815 webserver.cc:460] Webserver started at http://127.0.98.2:35441/ using document root <none> and password file <none>
I20260927 11:19:40.890645 11815 fs_manager.cc:359] Metadata directory not provided
I20260927 11:19:40.890705 11815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:19:40.890884 11815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:19:40 dist-test-slave-lsvg krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:19:40.909799 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb49cac7c63b4ae299ad100c5857a687"
format_stamp: "Formatted at 2026-09-27 11:19:40 on dist-test-slave-lsvg"
server_key: "0127e90ef67337c90655a1a426d4fa78"
server_key_iv: "1575053ed5dee94fe165cb65b9294ab3"
server_key_version: "kuduclusterkey@0"
I20260927 11:19:40.910284 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb49cac7c63b4ae299ad100c5857a687"
format_stamp: "Formatted at 2026-09-27 11:19:40 on dist-test-slave-lsvg"
server_key: "0127e90ef67337c90655a1a426d4fa78"
server_key_iv: "1575053ed5dee94fe165cb65b9294ab3"
server_key_version: "kuduclusterkey@0"
I20260927 11:19:40.917251 11815 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
Sep 27 11:19:40 dist-test-slave-lsvg krb5kdc[11121](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790507980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:19:40.930617 11832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:19:40.930970 11815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 11:19:40.931084 11815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "eb49cac7c63b4ae299ad100c5857a687"
format_stamp: "Formatted at 2026-09-27 11:19:40 on dist-test-slave-lsvg"
server_key: "0127e90ef67337c90655a1a426d4fa78"
server_key_iv: "1575053ed5dee94fe165cb65b9294ab3"
server_key_version: "kuduclusterkey@0"
I20260927 11:19:40.931205 11815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:19:40.962785 11815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:19:40.965831 11815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:19:40.965986 11815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:19:40.966256 11815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:19:40.966697 11815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:19:40.966771 11815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:19:40.966835 11815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:19:40.966885 11815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:19:40.975237 11815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44243
I20260927 11:19:40.975311 11945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44243 every 8 connection(s)
I20260927 11:19:40.975776 11815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:19:40.982882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 11815
I20260927 11:19:40.982991 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
Sep 27 11:19:40 dist-test-slave-lsvg krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:19:40.983180 392 external_mini_cluster.cc:1428] Setting key 2b0dc324dc591de32c7f8b8e0cfed052
I20260927 11:19:40.987303 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:40.976550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45023 (local address 127.0.98.62:34233)
0927 11:19:40.976620 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 11:19:40.976622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:19:40.976886 (+ 264us) server_negotiation.cc:407] Connection header received
0927 11:19:40.977662 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:19:40.977664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:19:40.977677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:19:40.977714 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:19:40.978548 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:40.979321 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:40.980812 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:40.981017 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:40.981164 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:40.983566 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:19:40.983568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:19:40.983568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:19:40.983576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:19:40.985555 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:19:40.986093 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:19:40.986095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:19:40.986095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:19:40.986164 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:19:40.986472 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:19:40.986473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:19:40.986474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:19:40.986613 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:19:40.986677 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 11:19:40.987098 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 11:19:40.987198 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:19:40.988109 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:40.976553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34233 (local address 127.0.98.2:45023)
0927 11:19:40.976739 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 11:19:40.976787 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 11:19:40.977519 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:19:40.977905 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:19:40.977909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:19:40.978163 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:19:40.978432 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:40.978436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:40.979446 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:40.979446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:40.980709 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:40.980712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:40.981175 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:40.981176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:40.981318 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:40.981882 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:19:40.981890 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:19:40.983424 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:19:40.985672 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:19:40.985678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:19:40.985681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:19:40.985988 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:19:40.986247 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:19:40.986248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:19:40.986248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:19:40.986385 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:19:40.986733 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:19:40.986739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:19:40.986915 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 11:19:40.987703 (+ 788us) client_negotiation.cc:240] Negotiation successful
0927 11:19:40.987873 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260927 11:19:40.988852 11946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34233
I20260927 11:19:40.988945 11946 heartbeater.cc:464] Registering TS with master...
I20260927 11:19:40.989238 11946 heartbeater.cc:511] Master 127.0.98.62:34233 requested a full tablet report, sending...
I20260927 11:19:40.989674 11596 ts_manager.cc:194] Registered new tserver with Master: eb49cac7c63b4ae299ad100c5857a687 (127.0.98.2:44243)
I20260927 11:19:40.990134 11596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45023
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:19:41.015568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51743/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34233
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:19:41.132315 11952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:19:41.132673 11952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:19:41.132773 11952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:19:41.134558 11952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:19:41.134688 11952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:19:41.134799 11952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:19:41.136997 11952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:51743/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:19:41.137562 11952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:19:41.137897 11952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:19:41.144843 11958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:19:41.145210 11952 server_base.cc:1034] running on GCE node
W20260927 11:19:41.145205 11957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:19:41.145493 11960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:19:41.149330 11952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:19:41.149612 11952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:19:41.150781 11952 hybrid_clock.cc:648] HybridClock initialized: now 1790507981150757 us; error 41 us; skew 500 ppm
Sep 27 11:19:41 dist-test-slave-lsvg krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:19:41.154327 11952 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:19:41.154886 11952 webserver.cc:460] Webserver started at http://127.0.98.3:41145/ using document root <none> and password file <none>
I20260927 11:19:41.155169 11952 fs_manager.cc:359] Metadata directory not provided
I20260927 11:19:41.155236 11952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:19:41.155409 11952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:19:41 dist-test-slave-lsvg krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:19:41.172571 11952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "8a60be373c794222b38f5397f87ffd80"
format_stamp: "Formatted at 2026-09-27 11:19:41 on dist-test-slave-lsvg"
server_key: "996e0d05af45f34ee9081c7cd1a28936"
server_key_iv: "b8604af0ed3f962b5ca790c581fdd4d7"
server_key_version: "kuduclusterkey@0"
I20260927 11:19:41.173002 11952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8a60be373c794222b38f5397f87ffd80"
format_stamp: "Formatted at 2026-09-27 11:19:41 on dist-test-slave-lsvg"
server_key: "996e0d05af45f34ee9081c7cd1a28936"
server_key_iv: "b8604af0ed3f962b5ca790c581fdd4d7"
server_key_version: "kuduclusterkey@0"
I20260927 11:19:41.177608 11952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260927 11:19:41.191413 11967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:19:41.191676 11952 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:19:41.191790 11952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "8a60be373c794222b38f5397f87ffd80"
format_stamp: "Formatted at 2026-09-27 11:19:41 on dist-test-slave-lsvg"
server_key: "996e0d05af45f34ee9081c7cd1a28936"
server_key_iv: "b8604af0ed3f962b5ca790c581fdd4d7"
server_key_version: "kuduclusterkey@0"
I20260927 11:19:41.191919 11952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:19:41.229290 11952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:19:41.232564 11952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:19:41.232726 11952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:19:41.232999 11952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:19:41.233836 11952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:19:41.233897 11952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:19:41.233969 11952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:19:41.233999 11952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:19:41.241240 11952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46007
I20260927 11:19:41.241739 11952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:19:41.242947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 11952
I20260927 11:19:41.243055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:19:41.243234 392 external_mini_cluster.cc:1428] Setting key b344272f856fd964c3223656fb88a31c
Sep 27 11:19:41 dist-test-slave-lsvg krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:19:41.252080 12080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46007 every 8 connection(s)
I20260927 11:19:41.254330 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.242552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34233 (local address 127.0.98.3:49319)
0927 11:19:41.242727 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 11:19:41.242768 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 11:19:41.243478 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:19:41.244058 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:19:41.244063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:19:41.244276 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:19:41.244470 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:41.244475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.245348 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.245349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:41.246180 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:41.246182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.246567 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.246568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:41.246727 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.247287 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:19:41.247296 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:19:41.248633 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:19:41.250945 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:19:41.250952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:19:41.250955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:19:41.251230 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:19:41.253147 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:19:41.253148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:19:41.253148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:19:41.253291 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:19:41.253644 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:19:41.253649 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:19:41.253831 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 11:19:41.253946 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 11:19:41.254101 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":110,"threads_started":1}
I20260927 11:19:41.254887 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.242524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49319 (local address 127.0.98.62:34233)
0927 11:19:41.242571 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:19:41.242574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:19:41.242857 (+ 283us) server_negotiation.cc:407] Connection header received
0927 11:19:41.243650 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:19:41.243651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:19:41.243662 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:19:41.243694 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:19:41.244570 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.245256 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.246263 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.246478 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.246560 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.248723 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:19:41.248725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:19:41.248725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:19:41.248733 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:19:41.250839 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:19:41.252253 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:19:41.252254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:19:41.252255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:19:41.252330 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:19:41.253390 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:19:41.253391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:19:41.253392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:19:41.253549 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:19:41.253594 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 11:19:41.254001 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 11:19:41.254099 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:19:41.255508 12081 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34233
I20260927 11:19:41.255633 12081 heartbeater.cc:464] Registering TS with master...
I20260927 11:19:41.255832 12081 heartbeater.cc:511] Master 127.0.98.62:34233 requested a full tablet report, sending...
I20260927 11:19:41.256270 11596 ts_manager.cc:194] Registered new tserver with Master: 8a60be373c794222b38f5397f87ffd80 (127.0.98.3:46007)
I20260927 11:19:41.256743 11596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49319
I20260927 11:19:41.265612 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:19:41.275837 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.267371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.0.98.62:34233)
0927 11:19:41.267419 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:19:41.267422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:19:41.267585 (+ 163us) server_negotiation.cc:407] Connection header received
0927 11:19:41.267657 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:19:41.267658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:19:41.267677 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:19:41.267741 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:19:41.268496 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.269314 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.270369 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.270788 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.270864 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.272459 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:19:41.272461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:19:41.272461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:19:41.272469 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:19:41.274446 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:19:41.274870 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:19:41.274871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:19:41.274871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:19:41.274942 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:19:41.275192 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:19:41.275193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:19:41.275194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:19:41.275333 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:19:41.275408 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:19:41.275640 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 11:19:41.275731 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:19:41.277830 11596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:19:41.280220 11596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:19:41.290634 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.283203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39045 (local address 127.0.0.1:36912)
0927 11:19:41.283736 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 11:19:41.283777 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 11:19:41.284185 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:19:41.286914 (+ 2729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:19:41.286915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:19:41.286917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:19:41.286979 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:41.286980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.288257 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.288258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:41.289376 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:41.289378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.290366 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.290367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:41.290439 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.290443 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:19:41.290493 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 11:19:41.290554 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":338,"threads_started":1}
I20260927 11:19:41.290629 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.283375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.0.98.2:44243)
0927 11:19:41.283567 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 11:19:41.283573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:19:41.285134 (+ 1561us) server_negotiation.cc:407] Connection header received
0927 11:19:41.285276 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:19:41.285280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:19:41.285298 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:19:41.285391 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:19:41.285746 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.287153 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.289021 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.289846 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.289905 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.289986 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:19:41.290485 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 11:19:41.290531 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":88,"threads_started":1}
I20260927 11:19:41.291276 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.283217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44243 (local address 127.0.0.1:33220)
0927 11:19:41.285035 (+ 1818us) negotiation.cc:107] Waiting for socket to connect
0927 11:19:41.285066 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 11:19:41.285164 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:19:41.285501 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:19:41.285504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:19:41.285507 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:19:41.285587 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:41.285588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.287270 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.287271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:41.288879 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:41.288882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.290069 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.290070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:41.290181 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.290198 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 11:19:41.290780 (+ 582us) client_negotiation.cc:240] Negotiation successful
0927 11:19:41.290831 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1802,"thread_start_us":453,"threads_started":1}
I20260927 11:19:41.291314 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.283374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.0.98.1:39045)
0927 11:19:41.286669 (+ 3295us) server_negotiation.cc:206] Beginning negotiation
0927 11:19:41.286674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:19:41.286686 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:19:41.286719 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:19:41.286721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:19:41.286737 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:19:41.286838 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:19:41.287072 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.288154 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.289471 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.290216 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.290284 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.290333 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:19:41.291159 (+ 826us) server_negotiation.cc:299] Negotiation successful
0927 11:19:41.291220 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3260,"thread_start_us":51,"threads_started":1}
I20260927 11:19:41.291658 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.283771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46007 (local address 127.0.0.1:56732)
0927 11:19:41.285264 (+ 1493us) negotiation.cc:107] Waiting for socket to connect
0927 11:19:41.285272 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:19:41.285312 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:19:41.285575 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:19:41.285576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:19:41.285578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:19:41.285627 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:41.285629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.289281 (+ 3652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.289282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:41.290366 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:41.290367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.291430 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.291430 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:41.291487 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.291492 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:19:41.291540 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 11:19:41.291595 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1479,"thread_start_us":68,"threads_started":1}
I20260927 11:19:41.291872 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.283982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.0.98.3:46007)
0927 11:19:41.284977 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0927 11:19:41.284982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:19:41.285367 (+ 385us) server_negotiation.cc:407] Connection header received
0927 11:19:41.285409 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:19:41.285412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:19:41.285427 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:19:41.285496 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:19:41.285717 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.288304 (+ 2587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.290491 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.291252 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.291318 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.291394 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:19:41.291763 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 11:19:41.291802 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":66,"threads_started":1}
I20260927 11:19:41.292035 11733 tablet_service.cc:1467] Processing CreateTablet for tablet 9f649d98443845ec8f5a5da984ce2df2 (DEFAULT_TABLE table=test-table [id=857dfa680b974598a25af03ed34c86b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:19:41.292338 11733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f649d98443845ec8f5a5da984ce2df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:19:41.292276 11862 tablet_service.cc:1467] Processing CreateTablet for tablet 9f649d98443845ec8f5a5da984ce2df2 (DEFAULT_TABLE table=test-table [id=857dfa680b974598a25af03ed34c86b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:19:41.292637 11862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f649d98443845ec8f5a5da984ce2df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:19:41.293743 12015 tablet_service.cc:1467] Processing CreateTablet for tablet 9f649d98443845ec8f5a5da984ce2df2 (DEFAULT_TABLE table=test-table [id=857dfa680b974598a25af03ed34c86b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:19:41.294056 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f649d98443845ec8f5a5da984ce2df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:19:41.295768 12100 tablet_bootstrap.cc:492] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92: Bootstrap starting.
I20260927 11:19:41.296521 12100 tablet_bootstrap.cc:654] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92: Neither blocks nor log segments found. Creating new log.
I20260927 11:19:41.297402 12100 log.cc:824] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92: Log is configured to *not* fsync() on all Append() calls
I20260927 11:19:41.298220 12100 tablet_bootstrap.cc:492] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92: No bootstrap required, opened a new log
I20260927 11:19:41.298286 12100 ts_tablet_manager.cc:1397] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 11:19:41.300765 12100 raft_consensus.cc:348] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378a87abb750424c9427bc6438fe4a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39045 } } peers { permanent_uuid: "eb49cac7c63b4ae299ad100c5857a687" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44243 } } peers { permanent_uuid: "8a60be373c794222b38f5397f87ffd80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46007 } }
I20260927 11:19:41.300925 12100 raft_consensus.cc:374] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:19:41.300957 12100 raft_consensus.cc:729] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378a87abb750424c9427bc6438fe4a92, State: Initialized, Role: FOLLOWER
I20260927 11:19:41.301334 12100 consensus_queue.cc:260] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378a87abb750424c9427bc6438fe4a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39045 } } peers { permanent_uuid: "eb49cac7c63b4ae299ad100c5857a687" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44243 } } peers { permanent_uuid: "8a60be373c794222b38f5397f87ffd80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46007 } }
I20260927 11:19:41.301704 12100 ts_tablet_manager.cc:1428] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 11:19:41.301873 11807 heartbeater.cc:503] Master 127.0.98.62:34233 was elected leader, sending a full tablet report...
I20260927 11:19:41.302389 12102 tablet_bootstrap.cc:492] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687: Bootstrap starting.
I20260927 11:19:41.302590 12103 tablet_bootstrap.cc:492] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80: Bootstrap starting.
I20260927 11:19:41.304087 12103 tablet_bootstrap.cc:654] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80: Neither blocks nor log segments found. Creating new log.
I20260927 11:19:41.304880 12102 tablet_bootstrap.cc:654] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687: Neither blocks nor log segments found. Creating new log.
I20260927 11:19:41.305290 12103 log.cc:824] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80: Log is configured to *not* fsync() on all Append() calls
I20260927 11:19:41.305667 12102 log.cc:824] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687: Log is configured to *not* fsync() on all Append() calls
I20260927 11:19:41.306703 12103 tablet_bootstrap.cc:492] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80: No bootstrap required, opened a new log
I20260927 11:19:41.306783 12103 ts_tablet_manager.cc:1397] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 11:19:41.306907 12102 tablet_bootstrap.cc:492] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687: No bootstrap required, opened a new log
I20260927 11:19:41.306965 12102 ts_tablet_manager.cc:1397] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 11:19:41.309409 12102 raft_consensus.cc:348] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378a87abb750424c9427bc6438fe4a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39045 } } peers { permanent_uuid: "eb49cac7c63b4ae299ad100c5857a687" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44243 } } peers { permanent_uuid: "8a60be373c794222b38f5397f87ffd80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46007 } }
I20260927 11:19:41.309569 12102 raft_consensus.cc:374] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:19:41.309602 12102 raft_consensus.cc:729] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb49cac7c63b4ae299ad100c5857a687, State: Initialized, Role: FOLLOWER
I20260927 11:19:41.309839 12102 consensus_queue.cc:260] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378a87abb750424c9427bc6438fe4a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39045 } } peers { permanent_uuid: "eb49cac7c63b4ae299ad100c5857a687" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44243 } } peers { permanent_uuid: "8a60be373c794222b38f5397f87ffd80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46007 } }
I20260927 11:19:41.310158 12102 ts_tablet_manager.cc:1428] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:19:41.310330 11946 heartbeater.cc:503] Master 127.0.98.62:34233 was elected leader, sending a full tablet report...
I20260927 11:19:41.311208 12103 raft_consensus.cc:348] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378a87abb750424c9427bc6438fe4a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39045 } } peers { permanent_uuid: "eb49cac7c63b4ae299ad100c5857a687" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44243 } } peers { permanent_uuid: "8a60be373c794222b38f5397f87ffd80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46007 } }
I20260927 11:19:41.311376 12103 raft_consensus.cc:374] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:19:41.311416 12103 raft_consensus.cc:729] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a60be373c794222b38f5397f87ffd80, State: Initialized, Role: FOLLOWER
I20260927 11:19:41.311534 12103 consensus_queue.cc:260] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378a87abb750424c9427bc6438fe4a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39045 } } peers { permanent_uuid: "eb49cac7c63b4ae299ad100c5857a687" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44243 } } peers { permanent_uuid: "8a60be373c794222b38f5397f87ffd80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46007 } }
I20260927 11:19:41.311864 12103 ts_tablet_manager.cc:1428] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260927 11:19:41.312038 12081 heartbeater.cc:503] Master 127.0.98.62:34233 was elected leader, sending a full tablet report...
Sep 27 11:19:41 dist-test-slave-lsvg krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 11:19:41.461534 11808 tablet.cc:2369] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:19:41.477684 11947 tablet.cc:2369] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:19:41.492527 12082 tablet.cc:2369] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:19:41.596870 12107 raft_consensus.cc:482] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:19:41.597024 12107 raft_consensus.cc:504] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378a87abb750424c9427bc6438fe4a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39045 } } peers { permanent_uuid: "eb49cac7c63b4ae299ad100c5857a687" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44243 } } peers { permanent_uuid: "8a60be373c794222b38f5397f87ffd80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46007 } }
I20260927 11:19:41.597445 12107 leader_election.cc:288] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 378a87abb750424c9427bc6438fe4a92 (127.0.98.1:39045), 8a60be373c794222b38f5397f87ffd80 (127.0.98.3:46007)
I20260927 11:19:41.602023 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.598036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45613 (local address 127.0.98.3:46007)
0927 11:19:41.598080 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:19:41.598083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:19:41.598538 (+ 455us) server_negotiation.cc:407] Connection header received
0927 11:19:41.598643 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:19:41.598644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:19:41.598669 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:19:41.598700 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:19:41.598913 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.599563 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.600978 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.601526 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.601578 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.601590 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:19:41.601904 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 11:19:41.601949 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:19:41.602489 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.597771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46007 (local address 127.0.98.2:45613)
0927 11:19:41.598480 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 11:19:41.598490 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:19:41.598567 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:19:41.598770 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:19:41.598771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:19:41.598773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:19:41.598841 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:41.598843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.599707 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.599708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:41.600866 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:41.600870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.601690 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.601691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:41.601762 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.601767 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:19:41.602356 (+ 589us) client_negotiation.cc:240] Negotiation successful
0927 11:19:41.602409 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":79,"threads_started":1}
I20260927 11:19:41.603012 12019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f649d98443845ec8f5a5da984ce2df2" candidate_uuid: "eb49cac7c63b4ae299ad100c5857a687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a60be373c794222b38f5397f87ffd80" is_pre_election: true
I20260927 11:19:41.603178 12019 raft_consensus.cc:2454] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb49cac7c63b4ae299ad100c5857a687 in term 0.
I20260927 11:19:41.603509 11833 leader_election.cc:302] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a60be373c794222b38f5397f87ffd80, eb49cac7c63b4ae299ad100c5857a687; no voters:
I20260927 11:19:41.603645 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.598389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52669 (local address 127.0.98.1:39045)
0927 11:19:41.598430 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:19:41.598432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:19:41.600386 (+ 1954us) server_negotiation.cc:407] Connection header received
0927 11:19:41.600493 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:19:41.600493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:19:41.600506 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:19:41.600534 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:19:41.600740 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.601417 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.602720 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.603262 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.603311 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.603320 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:19:41.603537 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 11:19:41.603577 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:19:41.603647 12107 raft_consensus.cc:2781] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:19:41.603700 12107 raft_consensus.cc:482] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:19:41.603724 12107 raft_consensus.cc:3037] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:19:41.603906 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.598221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39045 (local address 127.0.98.2:52669)
0927 11:19:41.600279 (+ 2058us) negotiation.cc:107] Waiting for socket to connect
0927 11:19:41.600292 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:19:41.600416 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:19:41.600591 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:19:41.600592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:19:41.600594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:19:41.600661 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:41.600662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.601519 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.601519 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:41.602600 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:19:41.602603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.603381 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.603382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:19:41.603450 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.603454 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:19:41.603776 (+ 322us) client_negotiation.cc:240] Negotiation successful
0927 11:19:41.603817 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2041,"thread_start_us":64,"threads_started":1}
I20260927 11:19:41.604192 11742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f649d98443845ec8f5a5da984ce2df2" candidate_uuid: "eb49cac7c63b4ae299ad100c5857a687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378a87abb750424c9427bc6438fe4a92" is_pre_election: true
I20260927 11:19:41.604328 11742 raft_consensus.cc:2454] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb49cac7c63b4ae299ad100c5857a687 in term 0.
I20260927 11:19:41.604638 12107 raft_consensus.cc:504] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378a87abb750424c9427bc6438fe4a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39045 } } peers { permanent_uuid: "eb49cac7c63b4ae299ad100c5857a687" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44243 } } peers { permanent_uuid: "8a60be373c794222b38f5397f87ffd80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46007 } }
I20260927 11:19:41.604789 12107 leader_election.cc:288] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [CANDIDATE]: Term 1 election: Requested vote from peers 378a87abb750424c9427bc6438fe4a92 (127.0.98.1:39045), 8a60be373c794222b38f5397f87ffd80 (127.0.98.3:46007)
I20260927 11:19:41.604950 11742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f649d98443845ec8f5a5da984ce2df2" candidate_uuid: "eb49cac7c63b4ae299ad100c5857a687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378a87abb750424c9427bc6438fe4a92"
I20260927 11:19:41.605036 11742 raft_consensus.cc:3037] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:19:41.605296 12019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f649d98443845ec8f5a5da984ce2df2" candidate_uuid: "eb49cac7c63b4ae299ad100c5857a687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a60be373c794222b38f5397f87ffd80"
I20260927 11:19:41.605373 12019 raft_consensus.cc:3037] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:19:41.605970 11742 raft_consensus.cc:2454] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb49cac7c63b4ae299ad100c5857a687 in term 1.
I20260927 11:19:41.606190 11836 leader_election.cc:302] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 378a87abb750424c9427bc6438fe4a92, eb49cac7c63b4ae299ad100c5857a687; no voters:
I20260927 11:19:41.606227 12019 raft_consensus.cc:2454] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb49cac7c63b4ae299ad100c5857a687 in term 1.
I20260927 11:19:41.606441 12107 raft_consensus.cc:2781] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:19:41.606611 12107 raft_consensus.cc:686] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [term 1 LEADER]: Becoming Leader. State: Replica: eb49cac7c63b4ae299ad100c5857a687, State: Running, Role: LEADER
I20260927 11:19:41.606714 12107 consensus_queue.cc:237] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378a87abb750424c9427bc6438fe4a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39045 } } peers { permanent_uuid: "eb49cac7c63b4ae299ad100c5857a687" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44243 } } peers { permanent_uuid: "8a60be373c794222b38f5397f87ffd80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46007 } }
I20260927 11:19:41.607519 11596 catalog_manager.cc:5495] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb49cac7c63b4ae299ad100c5857a687 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "eb49cac7c63b4ae299ad100c5857a687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378a87abb750424c9427bc6438fe4a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb49cac7c63b4ae299ad100c5857a687" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a60be373c794222b38f5397f87ffd80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46007 } health_report { overall_health: UNKNOWN } } }
Sep 27 11:19:41 dist-test-slave-lsvg krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:19:41.644546 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.641091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.0.98.2:44243)
0927 11:19:41.641284 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 11:19:41.641286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:19:41.641295 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:19:41.641338 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:19:41.641339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:19:41.641364 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:19:41.641431 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:19:41.641593 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.642367 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.643484 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:19:41.643813 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:19:41.643900 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:19:41.644087 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:19:41.644233 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:19:41.644314 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:19:41.644417 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 11:19:41.644444 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260927 11:19:41.646464 12019 raft_consensus.cc:1261] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80 [term 1 FOLLOWER]: Refusing update from remote peer eb49cac7c63b4ae299ad100c5857a687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:19:41.646569 11742 raft_consensus.cc:1261] T 9f649d98443845ec8f5a5da984ce2df2 P 378a87abb750424c9427bc6438fe4a92 [term 1 FOLLOWER]: Refusing update from remote peer eb49cac7c63b4ae299ad100c5857a687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:19:41.646950 12107 consensus_queue.cc:1035] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a60be373c794222b38f5397f87ffd80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:19:41.647106 12112 consensus_queue.cc:1035] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "378a87abb750424c9427bc6438fe4a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:19:41.652631 11597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:19:41.652678 12116 mvcc.cc:205] Tried to move back new op lower bound from 7333920692821254144 to 7333920692661682176. Current Snapshot: MvccSnapshot[applied={T|T < 7333920692821254144}]
I20260927 11:19:41.652750 11597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:19:41.653473 11597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1748b329a57442b78d0784298395659c: Sending DeleteTablet for 3 replicas of tablet 9f649d98443845ec8f5a5da984ce2df2
I20260927 11:19:41.654053 11733 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f649d98443845ec8f5a5da984ce2df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:19:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36912
I20260927 11:19:41.654266 11862 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f649d98443845ec8f5a5da984ce2df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:19:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33220
I20260927 11:19:41.654450 12015 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f649d98443845ec8f5a5da984ce2df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:19:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56732
I20260927 11:19:41.655010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 11678
I20260927 11:19:41.659888 12117 mvcc.cc:205] Tried to move back new op lower bound from 7333920692821254144 to 7333920692661682176. Current Snapshot: MvccSnapshot[applied={T|T < 7333920692821254144}]
I20260927 11:19:41.661898 12124 tablet_replica.cc:330] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687: stopping tablet replica
I20260927 11:19:41.662536 12125 tablet_replica.cc:330] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80: stopping tablet replica
I20260927 11:19:41.662734 12125 raft_consensus.cc:2229] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:19:41.663092 12124 raft_consensus.cc:2229] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:19:41.664369 12124 raft_consensus.cc:2258] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:19:41.664522 12125 raft_consensus.cc:2258] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:19:41.665143 12124 ts_tablet_manager.cc:1905] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:19:41.667582 12124 ts_tablet_manager.cc:1918] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:19:41.667876 12124 log.cc:1196] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/9f649d98443845ec8f5a5da984ce2df2
I20260927 11:19:41.668269 12124 ts_tablet_manager.cc:1939] T 9f649d98443845ec8f5a5da984ce2df2 P eb49cac7c63b4ae299ad100c5857a687: Deleting consensus metadata
W20260927 11:19:41.668406 11584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39045 (error 108)
I20260927 11:19:41.668581 12125 ts_tablet_manager.cc:1905] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:19:41.669329 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.669202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39045 (local address 127.0.0.1:36926)
0927 11:19:41.669252 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:19:41.669287 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 11:19:41.669740 11584 catalog_manager.cc:4584] TS 378a87abb750424c9427bc6438fe4a92 (127.0.98.1:39045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f649d98443845ec8f5a5da984ce2df2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39045: connect: Connection refused (error 111)
I20260927 11:19:41.669986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 11815
I20260927 11:19:41.671271 12125 ts_tablet_manager.cc:1918] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:19:41.671343 12125 log.cc:1196] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/9f649d98443845ec8f5a5da984ce2df2
I20260927 11:19:41.671947 12125 ts_tablet_manager.cc:1939] T 9f649d98443845ec8f5a5da984ce2df2 P 8a60be373c794222b38f5397f87ffd80: Deleting consensus metadata
I20260927 11:19:41.672371 11581 catalog_manager.cc:4841] TS 8a60be373c794222b38f5397f87ffd80 (127.0.98.3:46007): tablet 9f649d98443845ec8f5a5da984ce2df2 (table test-table [id=857dfa680b974598a25af03ed34c86b5]) successfully deleted
I20260927 11:19:41.676200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 11952
I20260927 11:19:41.676863 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:19:41.676240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44243 (local address 127.0.0.1:33228)
0927 11:19:41.676793 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 11:19:41.676824 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531}
I20260927 11:19:41.680728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 11565
I20260927 11:19:41.688088 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 11:19:42.032827 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 11:19:42.032932 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 11:19:42.377338 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 11:19:42.377 UTC [11151] LOG: received smart shutdown request
2026-09-27 11:19:42.380 UTC [11151] LOG: background worker "logical replication launcher" (PID 11160) exited with exit code 1
2026-09-27 11:19:42.381 UTC [11155] LOG: shutting down
2026-09-27 11:19:42.554 UTC [11151] LOG: database system is shut down
2026-09-27T11:19:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46934 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:19:42 dist-test-slave-lsvg krb5kdc[12137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:19:42 dist-test-slave-lsvg krb5kdc[12137](info): set up 2 sockets
Sep 27 11:19:42 dist-test-slave-lsvg krb5kdc[12137](info): commencing operation
krb5kdc: starting...
W20260927 11:19:43.701012 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:19:43 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:19:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 11:19:43.867949 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskuJOgQn/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/postgres -l logfile start
2026-09-27 11:19:46.401 UTC [12167] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 11:19:46.401 UTC [12167] LOG: listening on IPv4 address "127.0.98.20", port 54739
2026-09-27 11:19:46.405 UTC [12167] LOG: listening on Unix socket "/tmp/.s.PGSQL.54739"
2026-09-27 11:19:46.426 UTC [12170] LOG: database system was shut down at 2026-09-27 11:19:44 UTC
2026-09-27 11:19:46.429 UTC [12167] LOG: database system is ready to accept connections
I20260927 11:19:47.432979 392 mini_postgres.cc:96] Postgres bound to 54739
2026-09-27 11:19:47.439 UTC [12178] FATAL: database "slave" does not exist
127.0.98.20:54739 - accepting connections
I20260927 11:19:47.439682 392 mini_ranger.cc:161] Starting Ranger...
I20260927 11:19:47.455102 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 11:19:47.980433 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 11:19:47.980531 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-admin
2026-09-27 11:19:48,259 [I] DB FLAVOR :POSTGRES
2026-09-27 11:19:48,260 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 11:19:48,260 [I] Checking connection..
2026-09-27 11:19:48,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 11:19:48,510 [I] Checking connection passed.
2026-09-27 11:19:48,511 [I] --------- Verifying version history table ---------
2026-09-27 11:19:48,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 11:19:48,777 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 11:19:48,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 11:19:49,014 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 11:19:49,014 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 11:19:49,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 11:19:49.267 UTC [12242] WARNING: there is no transaction in progress
2026-09-27 11:19:49,273 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 11:19:49,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 11:19:49,518 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 11:19:49,519 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 11:19:49,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 11:19:49,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 11:19:50,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lsvg.c.gcp-upstream.internal','N') ;"
2026-09-27 11:19:50,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 11:19:50,518 [I] Table x_portal_user does not exist in database ranger
2026-09-27 11:19:50,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 11:19:50,768 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 11:19:50,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 11:19:51,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 11:19:51,269 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 11:19:51,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 11:19:52.272 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.289 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.308 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.326 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.343 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.363 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.425 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.433 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.449 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.460 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.471 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.486 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.496 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.503 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.511 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:52.520 UTC [12350] WARNING: there is no transaction in progress
2026-09-27 11:19:53,284 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 11:19:53,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 11:19:53,536 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 11:19:53,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 11:19:53,799 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 11:19:53,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 11:19:54,048 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 11:19:54,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 11:19:54,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 11:19:54,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lsvg.c.gcp-upstream.internal';"
2026-09-27 11:19:54,805 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 11:19:54,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 11:19:55,060 [I] DB_PATCHES have already been applied
I20260927 11:19:55.066453 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskuJOgQn/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 11:19:55.066521 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 11:19:55.069669 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:40575
Sep 27, 2026 11:19:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 11:19:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 11:19:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 11:19:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 11:19:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 11:19:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 11:19:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 11:19:56.203066 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.763s user 0.003s sys 0.007s
Sep 27, 2026 11:20:15 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 11:20:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 11:20:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 11:20:15 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 11:20:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 11:20:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 11:20:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 11:20:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 11:20:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 11:20:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 11:20:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 11:20:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 11:20:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 11:20:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 11:20:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 11:20:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 11:20:15 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 11:20:16 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 11:20:16.886729 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 11:20:16 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508016, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 11:20:16.975193 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 11:20:16.988925 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 11:20:17.935401 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 11:20:17.935503 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-kms
I20260927 11:20:17.935521 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54739
2026-09-27 11:20:18,098 [I] DB FLAVOR :POSTGRES
2026-09-27 11:20:18,098 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 11:20:18,098 [I] Checking connection
2026-09-27 11:20:18,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 11:20:18,329 [I] connection success
2026-09-27 11:20:18,329 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 11:20:18,329 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 11:20:18,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 11:20:18,573 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 11:20:18,573 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 11:20:18,574 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 11:20:18,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54739/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 11:20:18,842 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 11:20:19.324715 392 mini_ranger_kms.cc:310] Created kms service
I20260927 11:20:19.536324 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 11:20:19.582286 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 11:20:19 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508016, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:20:19.972047 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 11:20:19.972260 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskuJOgQn/build/release/bin/postgresql.jar:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskuJOgQn/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 11:20:19.972285 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 11:20:19.975716 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12530
I20260927 11:20:19.975804 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:49953
Sep 27, 2026 11:20:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 11:20:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 11:20:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 11:20:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 11:20:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 11:20:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 11:20:22.200794 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.226s user 0.006s sys 0.139s
I20260927 11:20:22.200892 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 11:20:22.200937 392 mini_ranger_kms.cc:401] 127.0.98.20:49953/kms/v1/keys
Sep 27 11:20:22 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790508022, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 11:20:22 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508022, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 11:20:23 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508022, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 11:20:25 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:20:25.363677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39235
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39235
--ranger_config_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49953/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:25.464465 12581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:25.464704 12581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:25.464726 12581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:25.465898 12581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:20:25.465940 12581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:25.465955 12581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:25.465968 12581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:20:25.465981 12581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:20:25.467556 12581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:49953/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39235
--ranger_config_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:25.467801 12581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:25.468040 12581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:25.474115 12587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:25.474252 12586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:25.474140 12581 server_base.cc:1034] running on GCE node
W20260927 11:20:25.474115 12589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:25.474912 12581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:25.475234 12581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:25.476421 12581 hybrid_clock.cc:648] HybridClock initialized: now 1790508025476403 us; error 32 us; skew 500 ppm
Sep 27 11:20:25 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:25.482417 12581 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:20:25.483042 12581 webserver.cc:460] Webserver started at http://127.0.98.62:42469/ using document root <none> and password file <none>
I20260927 11:20:25.483301 12581 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:25.483363 12581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:25.483527 12581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:20:25 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:20:25.537289 12581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "a9a6693c40954f83b2c272401c22dd5d"
format_stamp: "Formatted at 2026-09-27 11:20:25 on dist-test-slave-lsvg"
server_key: "5a599f6f70cd5074a7b5dc659c203165"
server_key_iv: "87e718e26e3a1276904b665001597a98"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:25.537760 12581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "a9a6693c40954f83b2c272401c22dd5d"
format_stamp: "Formatted at 2026-09-27 11:20:25 on dist-test-slave-lsvg"
server_key: "5a599f6f70cd5074a7b5dc659c203165"
server_key_iv: "87e718e26e3a1276904b665001597a98"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:25.539572 12581 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 11:20:25.552162 12596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:25.552421 12581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:25.552536 12581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "a9a6693c40954f83b2c272401c22dd5d"
format_stamp: "Formatted at 2026-09-27 11:20:25 on dist-test-slave-lsvg"
server_key: "5a599f6f70cd5074a7b5dc659c203165"
server_key_iv: "87e718e26e3a1276904b665001597a98"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:25.552663 12581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:25.566814 12581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:25.568787 12581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:25.568936 12581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:25.573954 12581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39235
I20260927 11:20:25.574041 12648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39235 every 8 connection(s)
I20260927 11:20:25.574414 12581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:20:25.575105 12649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:25.577541 12649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d: Bootstrap starting.
I20260927 11:20:25.578140 12649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:25.578429 12649 log.cc:824] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:25.579103 12649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d: No bootstrap required, opened a new log
I20260927 11:20:25.580127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 12581
I20260927 11:20:25.580229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:20:25.580413 392 external_mini_cluster.cc:1428] Setting key 7073b5455ae77a5e8d9ff64fb60a1b4f
I20260927 11:20:25.581712 12649 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a6693c40954f83b2c272401c22dd5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39235 } }
I20260927 11:20:25.581949 12649 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:25.582005 12649 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a6693c40954f83b2c272401c22dd5d, State: Initialized, Role: FOLLOWER
I20260927 11:20:25.582180 12649 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a6693c40954f83b2c272401c22dd5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39235 } }
I20260927 11:20:25.582286 12649 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:20:25.582343 12649 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:20:25.582412 12649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:25.583123 12649 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a6693c40954f83b2c272401c22dd5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39235 } }
I20260927 11:20:25.583231 12649 leader_election.cc:302] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9a6693c40954f83b2c272401c22dd5d; no voters:
I20260927 11:20:25.583417 12649 leader_election.cc:288] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:20:25.583477 12654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:20:25.583671 12654 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 1 LEADER]: Becoming Leader. State: Replica: a9a6693c40954f83b2c272401c22dd5d, State: Running, Role: LEADER
I20260927 11:20:25.583725 12649 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:20:25.583806 12654 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a6693c40954f83b2c272401c22dd5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39235 } }
I20260927 11:20:25.584183 12654 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9a6693c40954f83b2c272401c22dd5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a6693c40954f83b2c272401c22dd5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39235 } } }
I20260927 11:20:25.584326 12654 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:25.584515 12654 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9a6693c40954f83b2c272401c22dd5d. Latest consensus state: current_term: 1 leader_uuid: "a9a6693c40954f83b2c272401c22dd5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a6693c40954f83b2c272401c22dd5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39235 } } }
I20260927 11:20:25.584621 12654 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:25.584790 12663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:20:25.585356 12663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:20:25.585655 12649 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 27 11:20:25 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:25.590126 12671 mvcc.cc:205] Tried to move back new op lower bound from 7333920872812384256 to 7333920872791830528. Current Snapshot: MvccSnapshot[applied={T|T < 7333920872812384256}]
I20260927 11:20:25.594313 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:25.580947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.0.98.62:39235)
0927 11:20:25.581151 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:25.581165 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:25.581312 (+ 147us) server_negotiation.cc:407] Connection header received
0927 11:20:25.581677 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:25.581685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:25.581748 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:25.581893 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:25.582653 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:25.584036 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:25.585284 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:25.585905 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:25.588088 (+ 2183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:25.589283 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:25.589289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:25.589289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:25.589310 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:25.592193 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:25.592615 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:25.592618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:25.592619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:25.592723 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:25.592941 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:25.592942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:25.592942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:25.593275 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:25.593860 (+ 585us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:25.593931 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 11:20:25.594066 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":108,"threads_started":1}
I20260927 11:20:25.595911 12663 catalog_manager.cc:1347] Generated new cluster ID: eecd24e8c7534e20ad1fe6abaf3c36e0
I20260927 11:20:25.595956 12663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:20:25.606292 12663 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:20:25.607141 12663 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:20:25.620210 12663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d: Generated new TSK 0
I20260927 11:20:25.620572 12663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:20:26.404681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39235
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:26.491562 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:26.491794 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:26.491835 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:26.493036 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:26.493150 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:26.493197 12694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:20:26.494827 12694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:49953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:26.495100 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:26.495337 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:26.500926 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:26.500975 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:26.501251 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:26.501756 12694 server_base.cc:1034] running on GCE node
I20260927 11:20:26.501987 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:26.502256 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:26.503417 12694 hybrid_clock.cc:648] HybridClock initialized: now 1790508026503399 us; error 37 us; skew 500 ppm
Sep 27 11:20:26 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:26.506105 12694 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:20:26.506623 12694 webserver.cc:460] Webserver started at http://127.0.98.1:46617/ using document root <none> and password file <none>
I20260927 11:20:26.506855 12694 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:26.506924 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:26.507071 12694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:20:26 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:20:26.520241 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "da78c16a4f4247eabd2d9be3dd0ed162"
format_stamp: "Formatted at 2026-09-27 11:20:26 on dist-test-slave-lsvg"
server_key: "c532eb59985fd96bcfae6c81df637841"
server_key_iv: "df6879dca348e8fc80088d7f89d4a44b"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:26.520851 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "da78c16a4f4247eabd2d9be3dd0ed162"
format_stamp: "Formatted at 2026-09-27 11:20:26 on dist-test-slave-lsvg"
server_key: "c532eb59985fd96bcfae6c81df637841"
server_key_iv: "df6879dca348e8fc80088d7f89d4a44b"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:26.522928 12694 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 11:20:26.538153 12709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:26.538420 12694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 11:20:26.538667 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "da78c16a4f4247eabd2d9be3dd0ed162"
format_stamp: "Formatted at 2026-09-27 11:20:26 on dist-test-slave-lsvg"
server_key: "c532eb59985fd96bcfae6c81df637841"
server_key_iv: "df6879dca348e8fc80088d7f89d4a44b"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:26.538786 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:26.566509 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:26.569391 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:26.569564 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:26.569852 12694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:26.570305 12694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:26.570444 12694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:26.570567 12694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:26.570659 12694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:26.578507 12694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46823
I20260927 11:20:26.578931 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:20:26.579792 12824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46823 every 8 connection(s)
I20260927 11:20:26.580425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 12694
I20260927 11:20:26.580525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:20:26.580716 392 external_mini_cluster.cc:1428] Setting key ef18c173b275f341e58446abf549526b
Sep 27 11:20:26 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:26.592845 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:26.580740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39235 (local address 127.0.98.1:41405)
0927 11:20:26.581247 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:26.581296 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:26.581948 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:26.582379 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:26.582384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:26.582649 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:26.582948 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:26.582955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:26.584070 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:26.584071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:26.585406 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:26.585415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:26.586048 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:26.586049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:26.586152 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:26.586764 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:26.586783 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:26.588465 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:26.590844 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:26.590851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:26.590854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:26.591228 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:26.591476 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:26.591477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:26.591478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:26.591624 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:26.591987 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:26.591994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:26.592206 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 11:20:26.592336 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 11:20:26.592523 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":90,"threads_started":1}
I20260927 11:20:26.593554 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:26.581083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41405 (local address 127.0.98.62:39235)
0927 11:20:26.582134 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:26.582136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:26.582144 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:20:26.582167 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:26.582168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:26.582180 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:26.582247 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:26.583067 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:26.583952 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:26.585534 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:26.585865 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:26.585958 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:26.588581 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:26.588583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:26.588583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:26.588592 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:26.590736 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:26.591324 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:26.591326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:26.591327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:26.591398 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:26.591705 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:26.591706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:26.591706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:26.591865 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:26.591953 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:26.593323 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0927 11:20:26.593421 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":77,"threads_started":1}
I20260927 11:20:26.594427 12825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39235
I20260927 11:20:26.594530 12825 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:26.594726 12825 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
I20260927 11:20:26.595283 12613 ts_manager.cc:194] Registered new tserver with Master: da78c16a4f4247eabd2d9be3dd0ed162 (127.0.98.1:46823)
I20260927 11:20:26.596233 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41405
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:20:26.632398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39235
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:26.749495 12831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:26.749852 12831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:26.749956 12831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:26.751784 12831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:26.752004 12831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:26.752125 12831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:20:26.754427 12831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:49953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:26.754913 12831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:26.755220 12831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:26.761786 12839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:26.762197 12841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:26.763307 12831 server_base.cc:1034] running on GCE node
W20260927 11:20:26.763486 12838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:26.763751 12831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:26.764014 12831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:26.765133 12831 hybrid_clock.cc:648] HybridClock initialized: now 1790508026765107 us; error 41 us; skew 500 ppm
Sep 27 11:20:26 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:26.768373 12831 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:20:26.768874 12831 webserver.cc:460] Webserver started at http://127.0.98.2:34017/ using document root <none> and password file <none>
I20260927 11:20:26.769222 12831 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:26.769450 12831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:26.769622 12831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:20:26 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:20:26.786450 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "72c77f4f0cec454d96ef056c3b2ec1fd"
format_stamp: "Formatted at 2026-09-27 11:20:26 on dist-test-slave-lsvg"
server_key: "2b2e277a7e37867e643220c7c1bd7cc0"
server_key_iv: "fde349e5f1a445b0338f441d82944ecb"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:26.786909 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "72c77f4f0cec454d96ef056c3b2ec1fd"
format_stamp: "Formatted at 2026-09-27 11:20:26 on dist-test-slave-lsvg"
server_key: "2b2e277a7e37867e643220c7c1bd7cc0"
server_key_iv: "fde349e5f1a445b0338f441d82944ecb"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:26.790330 12831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
Sep 27 11:20:26 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790508026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:26.806910 12848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:26.807297 12831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:26.807410 12831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "72c77f4f0cec454d96ef056c3b2ec1fd"
format_stamp: "Formatted at 2026-09-27 11:20:26 on dist-test-slave-lsvg"
server_key: "2b2e277a7e37867e643220c7c1bd7cc0"
server_key_iv: "fde349e5f1a445b0338f441d82944ecb"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:26.807554 12831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:26.832253 12831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:26.843109 12831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:26.843334 12831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:26.843616 12831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:26.843990 12831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:26.844066 12831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:26.844134 12831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:26.844188 12831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:26.850416 12831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35271
I20260927 11:20:26.850930 12831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:20:26.853173 12961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35271 every 8 connection(s)
I20260927 11:20:26.860581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 12831
I20260927 11:20:26.860714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:20:26.861148 392 external_mini_cluster.cc:1428] Setting key 01040d50541dac544e180aedeb9756ea
Sep 27 11:20:26 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 11:20:26.878764 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:26.855051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48011 (local address 127.0.98.62:39235)
0927 11:20:26.855104 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:26.855106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:26.855534 (+ 428us) server_negotiation.cc:407] Connection header received
0927 11:20:26.856247 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:26.856248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:26.856259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:26.856290 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:26.857121 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:26.857927 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:26.859405 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:26.859737 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:26.859776 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:26.875107 (+ 15331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:26.875111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:26.875111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:26.875121 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:26.876971 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:26.877579 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:26.877580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:26.877580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:26.877642 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:26.877964 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:26.877965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:26.877965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:26.878112 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:26.878163 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:26.878556 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 11:20:26.878649 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:20:26.879735 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:26.854792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39235 (local address 127.0.98.2:48011)
0927 11:20:26.855409 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:26.855454 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:26.856098 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:26.856471 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:26.856475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:26.856727 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:26.856993 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:26.856998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:26.858035 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:26.858036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:26.859295 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:26.859307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:26.859871 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:26.859872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:26.860003 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:26.861119 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:26.861130 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:26.873128 (+ 11998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:26.877120 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:26.877131 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:26.877134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:26.877472 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:26.877728 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:26.877729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:26.877729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:26.877889 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:26.878198 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:26.878205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:26.878397 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 11:20:26.879284 (+ 887us) client_negotiation.cc:240] Negotiation successful
0927 11:20:26.879469 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":80,"threads_started":1}
I20260927 11:20:26.880462 12962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39235
I20260927 11:20:26.880558 12962 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:26.880759 12962 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
I20260927 11:20:26.881271 12613 ts_manager.cc:194] Registered new tserver with Master: 72c77f4f0cec454d96ef056c3b2ec1fd (127.0.98.2:35271)
I20260927 11:20:26.881803 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:20:26.896255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39235
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:27.013960 12968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:27.014359 12968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:27.014459 12968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:27.016276 12968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:27.016503 12968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:27.016613 12968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:20:27.018899 12968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:49953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:27.019376 12968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:27.019722 12968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:27.026535 12974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:27.026911 12973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:27.027143 12976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:27.028059 12968 server_base.cc:1034] running on GCE node
I20260927 11:20:27.028357 12968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:27.028699 12968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:27.029933 12968 hybrid_clock.cc:648] HybridClock initialized: now 1790508027029913 us; error 32 us; skew 500 ppm
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:27.033350 12968 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:20:27.033869 12968 webserver.cc:460] Webserver started at http://127.0.98.3:36741/ using document root <none> and password file <none>
I20260927 11:20:27.034111 12968 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:27.034158 12968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:27.034302 12968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:20:27.049532 12968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "be8abbc2c5fa4c0396d5f86e2cca376c"
format_stamp: "Formatted at 2026-09-27 11:20:27 on dist-test-slave-lsvg"
server_key: "c0425f2c7796306030d8a9940a9429a5"
server_key_iv: "b68f17496385e1d01286ca8fc2a454cd"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:27.049990 12968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "be8abbc2c5fa4c0396d5f86e2cca376c"
format_stamp: "Formatted at 2026-09-27 11:20:27 on dist-test-slave-lsvg"
server_key: "c0425f2c7796306030d8a9940a9429a5"
server_key_iv: "b68f17496385e1d01286ca8fc2a454cd"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:27.052109 12968 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 11:20:27.065371 12983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.065898 12968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:27.066128 12968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "be8abbc2c5fa4c0396d5f86e2cca376c"
format_stamp: "Formatted at 2026-09-27 11:20:27 on dist-test-slave-lsvg"
server_key: "c0425f2c7796306030d8a9940a9429a5"
server_key_iv: "b68f17496385e1d01286ca8fc2a454cd"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:27.066326 12968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:27.082607 12968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:27.085124 12968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:27.085268 12968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:27.085534 12968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:27.085915 12968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:27.085960 12968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.085989 12968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:27.086006 12968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.093860 12968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36163
I20260927 11:20:27.094439 12968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:20:27.095746 13096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36163 every 8 connection(s)
I20260927 11:20:27.101274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 12968
I20260927 11:20:27.101414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:20:27.101639 392 external_mini_cluster.cc:1428] Setting key ea6875065dbc1a4a1af283be20be038f
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:27.109323 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:27.096017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59239 (local address 127.0.98.62:39235)
0927 11:20:27.096066 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:27.096068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:27.097440 (+ 1372us) server_negotiation.cc:407] Connection header received
0927 11:20:27.098139 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:27.098142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:27.098155 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:27.098186 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:27.099028 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.099973 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.101673 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.101979 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.102020 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:27.104618 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:27.104620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:27.104620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:27.104627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:27.106293 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:27.106770 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:27.106771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:27.106772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:27.106827 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:27.108523 (+ 1696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:27.108524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:27.108524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:27.108683 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:27.108728 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:27.109143 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 11:20:27.109236 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:20:27.110132 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:27.095701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39235 (local address 127.0.98.3:59239)
0927 11:20:27.097291 (+ 1590us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:27.097340 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:27.097997 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:27.098374 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:27.098378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:27.098617 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:27.098895 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:27.098902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.100104 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.100105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:27.101558 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:27.101561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.102181 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.102182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:27.102320 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:27.102939 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:27.102949 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:27.104512 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:27.106392 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:27.106398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:27.106400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:27.106684 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:27.106892 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:27.106892 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:27.106893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:27.107016 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:27.108777 (+ 1761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:27.108784 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:27.108963 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 11:20:27.109694 (+ 731us) client_negotiation.cc:240] Negotiation successful
0927 11:20:27.109858 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1542,"thread_start_us":97,"threads_started":1}
I20260927 11:20:27.110985 13097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39235
I20260927 11:20:27.111143 13097 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:27.111449 13097 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
I20260927 11:20:27.111825 12613 ts_manager.cc:194] Registered new tserver with Master: be8abbc2c5fa4c0396d5f86e2cca376c (127.0.98.3:36163)
I20260927 11:20:27.112310 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59239
I20260927 11:20:27.113724 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:20:27.114384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 12694
I20260927 11:20:27.121333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 12831
I20260927 11:20:27.126588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 12968
I20260927 11:20:27.132141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 12581
I20260927 11:20:27.139115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39235
--webserver_interface=127.0.98.62
--webserver_port=42469
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39235
--ranger_config_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49953/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49953/kms with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:27.221408 13100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:27.221614 13100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:27.221635 13100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:27.222795 13100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:20:27.222841 13100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:27.222854 13100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:27.222895 13100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:20:27.222923 13100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:20:27.224581 13100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49953/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39235
--ranger_config_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:27.224833 13100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:27.225085 13100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:27.229882 13108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:27.229863 13106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:27.229853 13105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:27.230147 13100 server_base.cc:1034] running on GCE node
I20260927 11:20:27.230372 13100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:27.230685 13100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:27.231882 13100 hybrid_clock.cc:648] HybridClock initialized: now 1790508027231873 us; error 38 us; skew 500 ppm
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:27.234756 13100 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:20:27.235275 13100 webserver.cc:460] Webserver started at http://127.0.98.62:42469/ using document root <none> and password file <none>
I20260927 11:20:27.235541 13100 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:27.235620 13100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:20:27.296793 13100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:27.297674 13116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.297860 13100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:20:27.297969 13100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "a9a6693c40954f83b2c272401c22dd5d"
format_stamp: "Formatted at 2026-09-27 11:20:25 on dist-test-slave-lsvg"
server_key: "5a599f6f70cd5074a7b5dc659c203165"
server_key_iv: "87e718e26e3a1276904b665001597a98"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:27.298419 13100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:27.328503 13100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:27.330228 13100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:27.330401 13100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:27.334648 13100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39235
I20260927 11:20:27.334693 13168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39235 every 8 connection(s)
I20260927 11:20:27.335055 13100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:20:27.335671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 13100
I20260927 11:20:27.335992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46823
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39235
--webserver_port=46617
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49953/kms with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:20:27.336687 13169 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:20:27.337707 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d: Bootstrap starting.
I20260927 11:20:27.345782 13169 log.cc:824] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:27.347563 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:20:27.347859 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d: Bootstrap complete.
I20260927 11:20:27.350354 13169 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a6693c40954f83b2c272401c22dd5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39235 } }
I20260927 11:20:27.350584 13169 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a6693c40954f83b2c272401c22dd5d, State: Initialized, Role: FOLLOWER
I20260927 11:20:27.350750 13169 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a6693c40954f83b2c272401c22dd5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39235 } }
I20260927 11:20:27.350822 13169 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:20:27.350880 13169 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:20:27.350929 13169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:20:27.351907 13169 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a6693c40954f83b2c272401c22dd5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39235 } }
I20260927 11:20:27.351992 13169 leader_election.cc:302] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9a6693c40954f83b2c272401c22dd5d; no voters:
I20260927 11:20:27.352216 13169 leader_election.cc:288] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 11:20:27.352248 13173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 11:20:27.352362 13173 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [term 2 LEADER]: Becoming Leader. State: Replica: a9a6693c40954f83b2c272401c22dd5d, State: Running, Role: LEADER
I20260927 11:20:27.352475 13169 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:20:27.352517 13173 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a6693c40954f83b2c272401c22dd5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39235 } }
I20260927 11:20:27.352835 13173 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9a6693c40954f83b2c272401c22dd5d. Latest consensus state: current_term: 2 leader_uuid: "a9a6693c40954f83b2c272401c22dd5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a6693c40954f83b2c272401c22dd5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39235 } } }
I20260927 11:20:27.352839 13175 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a9a6693c40954f83b2c272401c22dd5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a6693c40954f83b2c272401c22dd5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39235 } } }
I20260927 11:20:27.352964 13173 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:27.353006 13175 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:27.353297 13182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:20:27.353803 13182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:20:27.353924 13182 catalog_manager.cc:1259] Loaded cluster ID: eecd24e8c7534e20ad1fe6abaf3c36e0
I20260927 11:20:27.353960 13182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:20:27.354151 13169 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 11:20:27.357540 13182 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:20:27.357910 13182 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d: Loaded TSK: 0
I20260927 11:20:27.358181 13182 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 11:20:27.445000 13171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:27.445338 13171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:27.445381 13171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:27.447149 13171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:27.447260 13171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:27.447329 13171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:20:27.449569 13171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46617
--tserver_master_addrs=127.0.98.62:39235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:27.449812 13171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:27.450102 13171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:27.456490 13207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:27.456557 13209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:27.456884 13206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:27.457106 13171 server_base.cc:1034] running on GCE node
I20260927 11:20:27.457329 13171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:27.457604 13171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:27.458781 13171 hybrid_clock.cc:648] HybridClock initialized: now 1790508027458765 us; error 31 us; skew 500 ppm
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:27.462097 13171 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:20:27.462658 13171 webserver.cc:460] Webserver started at http://127.0.98.1:46617/ using document root <none> and password file <none>
I20260927 11:20:27.462970 13171 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:27.463115 13171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:20:27.501557 13171 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260927 11:20:27.502501 13217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.502717 13171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:27.502811 13171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "da78c16a4f4247eabd2d9be3dd0ed162"
format_stamp: "Formatted at 2026-09-27 11:20:26 on dist-test-slave-lsvg"
server_key: "c532eb59985fd96bcfae6c81df637841"
server_key_iv: "df6879dca348e8fc80088d7f89d4a44b"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:27.503314 13171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:27.514870 13171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:27.522938 13171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:27.523178 13171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:27.523521 13171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:27.523983 13171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:27.524080 13171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.524189 13171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:27.524286 13171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.531275 13171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46823
I20260927 11:20:27.531754 13171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:20:27.533731 13330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46823 every 8 connection(s)
I20260927 11:20:27.539884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 13171
I20260927 11:20:27.540184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35271
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39235
--webserver_port=34017
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49953/kms with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:27.549693 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:27.534640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39235 (local address 127.0.98.1:44669)
0927 11:20:27.535289 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:27.535338 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:27.535987 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:27.536786 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:27.536790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:27.537043 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:27.537356 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:27.537362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.538984 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.538985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:27.542225 (+ 3240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:27.542229 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.543188 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.543189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:27.543352 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:27.544001 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:27.544013 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:27.545645 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:27.547885 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:27.547892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:27.547894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:27.548114 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:27.548397 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:27.548399 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:27.548399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:27.548504 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:27.548911 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:27.548917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:27.549115 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 11:20:27.549235 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 11:20:27.549418 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":85,"threads_started":1}
I20260927 11:20:27.550258 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:27.535029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44669 (local address 127.0.98.62:39235)
0927 11:20:27.536208 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:27.536230 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:27.536271 (+ 41us) server_negotiation.cc:407] Connection header received
0927 11:20:27.536428 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:27.536436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:27.536496 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:27.536631 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:27.537483 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.538858 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.542327 (+ 3469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.542957 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.543084 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:27.545765 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:27.545770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:27.545770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:27.545782 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:27.547774 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:27.548221 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:27.548224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:27.548225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:27.548315 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:27.548581 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:27.548582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:27.548583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:27.548826 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:27.548920 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:27.549919 (+ 999us) server_negotiation.cc:299] Negotiation successful
0927 11:20:27.550048 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":81,"threads_started":1}
I20260927 11:20:27.550976 13331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39235
I20260927 11:20:27.551098 13331 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:27.551307 13331 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
W20260927 11:20:27.552193 13331 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39235 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 11:20:27.552282 13331 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:27.552354 13331 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
I20260927 11:20:27.552651 13331 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:27.552744 13331 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
W20260927 11:20:27.553036 13331 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 11:20:27.666306 13335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:27.666567 13335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:27.666631 13335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:27.668165 13335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:27.668241 13335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:27.668288 13335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:20:27.670043 13335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34017
--tserver_master_addrs=127.0.98.62:39235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:27.670337 13335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:27.670653 13335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:27.676091 13341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:27.676129 13340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:27.676766 13335 server_base.cc:1034] running on GCE node
W20260927 11:20:27.676955 13343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:27.677215 13335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:27.677536 13335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:27.678830 13335 hybrid_clock.cc:648] HybridClock initialized: now 1790508027678800 us; error 45 us; skew 500 ppm
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:27.681543 13335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:20:27.682035 13335 webserver.cc:460] Webserver started at http://127.0.98.2:34017/ using document root <none> and password file <none>
I20260927 11:20:27.682255 13335 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:27.682334 13335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:20:27.701488 13335 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260927 11:20:27.702299 13351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.702454 13335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:20:27.702535 13335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "72c77f4f0cec454d96ef056c3b2ec1fd"
format_stamp: "Formatted at 2026-09-27 11:20:26 on dist-test-slave-lsvg"
server_key: "2b2e277a7e37867e643220c7c1bd7cc0"
server_key_iv: "fde349e5f1a445b0338f441d82944ecb"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:27.702981 13335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:27.728116 13335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:27.730257 13335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:27.730413 13335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:27.730679 13335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:27.731052 13335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:27.731105 13335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.731145 13335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:27.731171 13335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.738637 13335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35271
I20260927 11:20:27.738742 13464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35271 every 8 connection(s)
I20260927 11:20:27.739069 13335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:27.748677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 13335
I20260927 11:20:27.748916 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:27.739864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43273 (local address 127.0.98.62:39235)
0927 11:20:27.739906 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:27.739908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:27.740167 (+ 259us) server_negotiation.cc:407] Connection header received
0927 11:20:27.740784 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:27.740785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:27.740797 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:27.740857 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:27.741481 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.742180 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.743184 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.743397 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.743482 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:27.745619 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:27.745621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:27.745622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:27.745631 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:27.747315 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:27.747748 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:27.747749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:27.747750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:27.747818 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:27.748093 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:27.748094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:27.748095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:27.748245 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:27.748327 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:27.748735 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 11:20:27.748819 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:20:27.749009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:49953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36163
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39235
--webserver_port=36741
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49953/kms with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:20:27.749351 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:27.739866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39235 (local address 127.0.98.2:43273)
0927 11:20:27.740047 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:27.740082 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:27.740669 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:27.740927 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:27.740931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:27.741195 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:27.741386 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:27.741390 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.742277 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.742278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:27.743076 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:27.743079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.743504 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.743504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:27.743639 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:27.744173 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:27.744181 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:27.745494 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:27.747415 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:27.747422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:27.747424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:27.747668 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:27.747893 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:27.747893 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:27.747894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:27.748015 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:27.748331 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:27.748337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:27.748560 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 11:20:27.748713 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 11:20:27.748901 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260927 11:20:27.751854 13465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39235
I20260927 11:20:27.751955 13465 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:27.752166 13465 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
W20260927 11:20:27.752843 13465 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39235 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 11:20:27.752933 13465 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:27.753005 13465 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
I20260927 11:20:27.753340 13465 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:27.753415 13465 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
W20260927 11:20:27.753686 13465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 11:20:27.867823 13468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:27.868173 13468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:27.868253 13468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:27.869836 13468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:27.869907 13468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:27.869951 13468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:20:27.871495 13468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:49953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36741
--tserver_master_addrs=127.0.98.62:39235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:27.871791 13468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:27.872043 13468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:27.877712 13474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:27.877671 13477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:27.877916 13468 server_base.cc:1034] running on GCE node
W20260927 11:20:27.877682 13475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:27.878202 13468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:27.878512 13468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:27.879705 13468 hybrid_clock.cc:648] HybridClock initialized: now 1790508027879701 us; error 40 us; skew 500 ppm
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:27.882711 13468 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:20:27.883270 13468 webserver.cc:460] Webserver started at http://127.0.98.3:36741/ using document root <none> and password file <none>
I20260927 11:20:27.883507 13468 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:27.883566 13468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:20:27.900686 13468 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:27.901583 13485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.901752 13468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.901852 13468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "be8abbc2c5fa4c0396d5f86e2cca376c"
format_stamp: "Formatted at 2026-09-27 11:20:27 on dist-test-slave-lsvg"
server_key: "c0425f2c7796306030d8a9940a9429a5"
server_key_iv: "b68f17496385e1d01286ca8fc2a454cd"
server_key_version: "kuduclusterkey@0"
I20260927 11:20:27.902235 13468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:27.926386 13468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:27.929184 13468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:27.929337 13468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:27.929601 13468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:27.929965 13468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:27.930013 13468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.930047 13468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:27.930078 13468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:27.936265 13468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36163
I20260927 11:20:27.936797 13468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:20:27.937314 13598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36163 every 8 connection(s)
I20260927 11:20:27.941488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 13468
Sep 27 11:20:27 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:27.947578 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:27.938007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60105 (local address 127.0.98.62:39235)
0927 11:20:27.938084 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:27.938087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:27.938250 (+ 163us) server_negotiation.cc:407] Connection header received
0927 11:20:27.938743 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:27.938744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:27.938756 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:27.938810 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:27.939388 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.939881 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.940893 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.941156 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.941242 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:27.943966 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:27.943967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:27.943968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:27.943976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:27.945805 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:27.946353 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:27.946354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:27.946355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:27.946421 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:27.946771 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:27.946772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:27.946773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:27.946910 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:27.946990 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:27.947376 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 11:20:27.947461 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 11:20:27.947783 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:27.937924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39235 (local address 127.0.98.3:60105)
0927 11:20:27.938110 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:27.938152 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:27.938632 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:27.938892 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:27.938896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:27.939100 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:27.939288 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:27.939293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.939989 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.939990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:27.940786 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:27.940788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.941263 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.941264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:27.941435 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:27.942090 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:27.942101 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:27.943864 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:27.945884 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:27.945892 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:27.945894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:27.946210 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:27.946508 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:27.946508 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:27.946509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:27.946681 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:27.946996 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:27.947003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:27.947201 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 11:20:27.947338 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 11:20:27.947517 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260927 11:20:27.948277 13599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39235
I20260927 11:20:27.948397 13599 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:27.948621 13599 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
W20260927 11:20:27.949373 13599 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39235 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 11:20:27.949446 13599 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:27.949512 13599 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
I20260927 11:20:27.949916 13599 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:27.949991 13599 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
W20260927 11:20:27.950313 13599 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 11:20:27.955965 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:27.942109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.0.98.62:39235)
0927 11:20:27.942252 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:27.942255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:27.942448 (+ 193us) server_negotiation.cc:407] Connection header received
0927 11:20:27.942478 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:27.942479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:27.942494 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:27.942595 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:27.943610 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.945184 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.947478 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:27.947832 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:27.947908 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:27.949743 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:27.949744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:27.949745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:27.949757 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:27.954625 (+ 4868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:27.955058 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:27.955059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:27.955060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:27.955130 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:27.955372 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:27.955372 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:27.955373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:27.955521 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:27.955706 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:27.955752 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 11:20:27.955857 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
Sep 27 11:20:28 dist-test-slave-lsvg krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790508028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:28.553243 13331 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:28.553431 13331 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
I20260927 11:20:28.553862 13133 ts_manager.cc:194] Registered new tserver with Master: da78c16a4f4247eabd2d9be3dd0ed162 (127.0.98.1:46823)
I20260927 11:20:28.554883 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44669
I20260927 11:20:28.753867 13465 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:28.754033 13465 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
I20260927 11:20:28.754341 13133 ts_manager.cc:194] Registered new tserver with Master: 72c77f4f0cec454d96ef056c3b2ec1fd (127.0.98.2:35271)
I20260927 11:20:28.754886 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43273
Sep 27 11:20:28 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:20:28.950475 13599 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:28.950632 13599 heartbeater.cc:511] Master 127.0.98.62:39235 requested a full tablet report, sending...
I20260927 11:20:28.950937 13133 ts_manager.cc:194] Registered new tserver with Master: be8abbc2c5fa4c0396d5f86e2cca376c (127.0.98.3:36163)
I20260927 11:20:28.951328 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60105
I20260927 11:20:28.960561 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:20:28.970731 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:28.961545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46424 (local address 127.0.98.62:39235)
0927 11:20:28.964449 (+ 2904us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:28.964451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:28.964459 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:20:28.964481 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:28.964481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:28.964492 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:28.964588 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:28.965209 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.965724 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.966446 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.966652 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.966732 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:28.967992 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:28.967993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:28.967994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:28.968001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:28.969543 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:28.969945 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:28.969946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:28.969947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:28.970005 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:28.970215 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:28.970215 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:28.970216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:28.970327 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:28.970429 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:28.970537 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 11:20:28.970619 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2880,"thread_start_us":67,"threads_started":1}
I20260927 11:20:28.972805 13133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:20:28.974576 13133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:20:28.985267 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:28.977680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.0.98.2:35271)
0927 11:20:28.977977 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:28.977982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:28.981287 (+ 3305us) server_negotiation.cc:407] Connection header received
0927 11:20:28.981390 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:28.981393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:28.981447 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:28.982222 (+ 775us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:28.982346 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.983178 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.984238 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.984748 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.984853 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:28.984935 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:28.985109 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 11:20:28.985158 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":48,"threads_started":1}
I20260927 11:20:28.985534 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:28.977417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35271 (local address 127.0.0.1:52262)
0927 11:20:28.981158 (+ 3741us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:28.981208 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:28.981295 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:28.981657 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:28.981661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:28.981664 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:28.981912 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:28.981916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.983279 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.983280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:28.984125 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:28.984128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.984855 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.984856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:28.984940 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:28.984955 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 11:20:28.985060 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 11:20:28.985456 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3706,"thread_start_us":89,"threads_started":1}
I20260927 11:20:28.986454 13399 tablet_service.cc:1467] Processing CreateTablet for tablet ed71ca5f2c3043ec8a2944aee2aa358a (DEFAULT_TABLE table=test-table [id=134fc5eae3ea44bcbc588659c92d3c44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:20:28.986842 13399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed71ca5f2c3043ec8a2944aee2aa358a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:28.990171 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:28.977733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.0.98.1:46823)
0927 11:20:28.978091 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:28.978096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:28.985785 (+ 7689us) server_negotiation.cc:407] Connection header received
0927 11:20:28.985917 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:28.985919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:28.985936 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:28.985994 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:28.986226 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.987232 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.988495 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.989320 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.989377 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:28.989452 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:28.990022 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 11:20:28.990081 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":51,"threads_started":1}
I20260927 11:20:28.990434 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:28.977417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46823 (local address 127.0.0.1:45258)
0927 11:20:28.985710 (+ 8293us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:28.985722 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:28.985818 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:28.986110 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:28.986111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:28.986112 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:28.986144 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:28.986145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.987347 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.987348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:28.988361 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:28.988364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.989847 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.989847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:28.989914 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:28.989917 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:20:28.990305 (+ 388us) client_negotiation.cc:240] Negotiation successful
0927 11:20:28.990356 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8277,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260927 11:20:28.991160 13632 tablet_bootstrap.cc:492] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd: Bootstrap starting.
I20260927 11:20:28.991199 13265 tablet_service.cc:1467] Processing CreateTablet for tablet ed71ca5f2c3043ec8a2944aee2aa358a (DEFAULT_TABLE table=test-table [id=134fc5eae3ea44bcbc588659c92d3c44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:20:28.991559 13265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed71ca5f2c3043ec8a2944aee2aa358a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:28.991868 13632 tablet_bootstrap.cc:654] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:28.992244 13632 log.cc:824] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:28.993188 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:28.977459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36163 (local address 127.0.0.1:40552)
0927 11:20:28.989124 (+ 11665us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:28.989136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:28.989202 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:28.989529 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:28.989530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:28.989532 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:28.989573 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:28.989574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.990917 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.990918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:28.991971 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:28.991974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.992903 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.992905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:28.992968 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:28.992973 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:20:28.993044 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:20:28.993115 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11643,"thread_start_us":75,"threads_started":1}
I20260927 11:20:28.993211 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:28.977815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.0.98.3:36163)
0927 11:20:28.978157 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:28.978161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:28.989271 (+ 11110us) server_negotiation.cc:407] Connection header received
0927 11:20:28.989324 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:28.989327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:28.989344 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:28.989413 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:28.989644 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.990819 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.992076 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:28.992778 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:28.992900 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:28.992980 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:28.993095 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 11:20:28.993136 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":55,"threads_started":1}
I20260927 11:20:28.994228 13533 tablet_service.cc:1467] Processing CreateTablet for tablet ed71ca5f2c3043ec8a2944aee2aa358a (DEFAULT_TABLE table=test-table [id=134fc5eae3ea44bcbc588659c92d3c44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:20:28.994557 13533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed71ca5f2c3043ec8a2944aee2aa358a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:28.995064 13634 tablet_bootstrap.cc:492] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162: Bootstrap starting.
I20260927 11:20:28.995779 13634 tablet_bootstrap.cc:654] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:28.996176 13634 log.cc:824] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:29.000828 13636 tablet_bootstrap.cc:492] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c: Bootstrap starting.
I20260927 11:20:29.001971 13636 tablet_bootstrap.cc:654] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:29.002584 13636 log.cc:824] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:29.005762 13632 tablet_bootstrap.cc:492] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd: No bootstrap required, opened a new log
I20260927 11:20:29.005859 13632 ts_tablet_manager.cc:1397] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20260927 11:20:29.007452 13634 tablet_bootstrap.cc:492] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162: No bootstrap required, opened a new log
I20260927 11:20:29.007555 13634 ts_tablet_manager.cc:1397] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260927 11:20:29.008671 13632 raft_consensus.cc:348] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da78c16a4f4247eabd2d9be3dd0ed162" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46823 } } peers { permanent_uuid: "72c77f4f0cec454d96ef056c3b2ec1fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35271 } } peers { permanent_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } }
I20260927 11:20:29.008873 13632 raft_consensus.cc:374] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:29.008919 13632 raft_consensus.cc:729] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c77f4f0cec454d96ef056c3b2ec1fd, State: Initialized, Role: FOLLOWER
I20260927 11:20:29.009076 13632 consensus_queue.cc:260] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da78c16a4f4247eabd2d9be3dd0ed162" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46823 } } peers { permanent_uuid: "72c77f4f0cec454d96ef056c3b2ec1fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35271 } } peers { permanent_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } }
I20260927 11:20:29.009380 13632 ts_tablet_manager.cc:1428] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 11:20:29.009431 13465 heartbeater.cc:503] Master 127.0.98.62:39235 was elected leader, sending a full tablet report...
I20260927 11:20:29.010190 13634 raft_consensus.cc:348] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da78c16a4f4247eabd2d9be3dd0ed162" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46823 } } peers { permanent_uuid: "72c77f4f0cec454d96ef056c3b2ec1fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35271 } } peers { permanent_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } }
I20260927 11:20:29.010370 13634 raft_consensus.cc:374] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:29.010416 13634 raft_consensus.cc:729] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da78c16a4f4247eabd2d9be3dd0ed162, State: Initialized, Role: FOLLOWER
I20260927 11:20:29.010633 13634 consensus_queue.cc:260] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da78c16a4f4247eabd2d9be3dd0ed162" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46823 } } peers { permanent_uuid: "72c77f4f0cec454d96ef056c3b2ec1fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35271 } } peers { permanent_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } }
I20260927 11:20:29.010982 13634 ts_tablet_manager.cc:1428] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 11:20:29.011134 13331 heartbeater.cc:503] Master 127.0.98.62:39235 was elected leader, sending a full tablet report...
I20260927 11:20:29.020470 13636 tablet_bootstrap.cc:492] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c: No bootstrap required, opened a new log
I20260927 11:20:29.020579 13636 ts_tablet_manager.cc:1397] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20260927 11:20:29.022660 13636 raft_consensus.cc:348] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da78c16a4f4247eabd2d9be3dd0ed162" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46823 } } peers { permanent_uuid: "72c77f4f0cec454d96ef056c3b2ec1fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35271 } } peers { permanent_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } }
I20260927 11:20:29.022835 13636 raft_consensus.cc:374] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:29.022879 13636 raft_consensus.cc:729] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8abbc2c5fa4c0396d5f86e2cca376c, State: Initialized, Role: FOLLOWER
I20260927 11:20:29.023023 13636 consensus_queue.cc:260] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da78c16a4f4247eabd2d9be3dd0ed162" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46823 } } peers { permanent_uuid: "72c77f4f0cec454d96ef056c3b2ec1fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35271 } } peers { permanent_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } }
I20260927 11:20:29.023378 13599 heartbeater.cc:503] Master 127.0.98.62:39235 was elected leader, sending a full tablet report...
I20260927 11:20:29.023370 13636 ts_tablet_manager.cc:1428] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
W20260927 11:20:29.033236 13332 tablet.cc:2369] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 11:20:29 dist-test-slave-lsvg krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:20:29.187953 13640 raft_consensus.cc:482] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:20:29.188107 13640 raft_consensus.cc:504] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da78c16a4f4247eabd2d9be3dd0ed162" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46823 } } peers { permanent_uuid: "72c77f4f0cec454d96ef056c3b2ec1fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35271 } } peers { permanent_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } }
I20260927 11:20:29.188524 13640 leader_election.cc:288] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da78c16a4f4247eabd2d9be3dd0ed162 (127.0.98.1:46823), 72c77f4f0cec454d96ef056c3b2ec1fd (127.0.98.2:35271)
W20260927 11:20:29.188517 13600 tablet.cc:2369] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:20:29.197885 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:29.188945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51907 (local address 127.0.98.1:46823)
0927 11:20:29.189131 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:29.189133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:29.193245 (+ 4112us) server_negotiation.cc:407] Connection header received
0927 11:20:29.193370 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:29.193371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:29.193386 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:29.193417 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:29.193695 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:29.194485 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:29.196767 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:29.197416 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:29.197466 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:29.197478 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:29.197757 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 11:20:29.197803 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260927 11:20:29.202199 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:29.189179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54761 (local address 127.0.98.2:35271)
0927 11:20:29.189220 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:29.189221 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:29.193244 (+ 4023us) server_negotiation.cc:407] Connection header received
0927 11:20:29.193335 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:29.193337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:29.193350 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:29.193402 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:29.195508 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:29.196247 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:29.201159 (+ 4912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:29.201775 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:29.201829 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:29.201840 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:29.202084 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 11:20:29.202130 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:20:29.202538 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:29.188912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35271 (local address 127.0.98.3:54761)
0927 11:20:29.193151 (+ 4239us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:29.193163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:29.193261 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:29.195302 (+ 2041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:29.195303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:29.195305 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:29.195390 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:29.195391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:29.196360 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:29.196360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:29.197539 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:29.197542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:29.201921 (+ 4379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:29.201922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:29.202004 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:29.202007 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:20:29.202355 (+ 348us) client_negotiation.cc:240] Negotiation successful
0927 11:20:29.202404 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4217,"thread_start_us":85,"threads_started":1}
I20260927 11:20:29.202885 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:29.188733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46823 (local address 127.0.98.3:51907)
0927 11:20:29.193143 (+ 4410us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:29.193159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:29.193278 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:29.193486 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:29.193487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:29.193489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:29.193589 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:29.193591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:29.195329 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:29.195331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:29.196644 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:29.196647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:29.197539 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:29.197540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:29.197612 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:29.197615 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:20:29.202228 (+ 4613us) client_negotiation.cc:240] Negotiation successful
0927 11:20:29.202801 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4386,"thread_start_us":97,"threads_started":1}
I20260927 11:20:29.202888 13406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed71ca5f2c3043ec8a2944aee2aa358a" candidate_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c77f4f0cec454d96ef056c3b2ec1fd" is_pre_election: true
I20260927 11:20:29.203065 13406 raft_consensus.cc:2454] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8abbc2c5fa4c0396d5f86e2cca376c in term 0.
I20260927 11:20:29.203325 13489 leader_election.cc:302] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72c77f4f0cec454d96ef056c3b2ec1fd, be8abbc2c5fa4c0396d5f86e2cca376c; no voters:
I20260927 11:20:29.203365 13285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed71ca5f2c3043ec8a2944aee2aa358a" candidate_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da78c16a4f4247eabd2d9be3dd0ed162" is_pre_election: true
I20260927 11:20:29.203508 13285 raft_consensus.cc:2454] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8abbc2c5fa4c0396d5f86e2cca376c in term 0.
I20260927 11:20:29.203502 13640 raft_consensus.cc:2781] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:20:29.203552 13640 raft_consensus.cc:482] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:20:29.203581 13640 raft_consensus.cc:3037] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:29.204579 13640 raft_consensus.cc:504] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da78c16a4f4247eabd2d9be3dd0ed162" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46823 } } peers { permanent_uuid: "72c77f4f0cec454d96ef056c3b2ec1fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35271 } } peers { permanent_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } }
I20260927 11:20:29.204715 13640 leader_election.cc:288] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [CANDIDATE]: Term 1 election: Requested vote from peers da78c16a4f4247eabd2d9be3dd0ed162 (127.0.98.1:46823), 72c77f4f0cec454d96ef056c3b2ec1fd (127.0.98.2:35271)
I20260927 11:20:29.205029 13285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed71ca5f2c3043ec8a2944aee2aa358a" candidate_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da78c16a4f4247eabd2d9be3dd0ed162"
I20260927 11:20:29.205129 13285 raft_consensus.cc:3037] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:29.206027 13285 raft_consensus.cc:2454] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be8abbc2c5fa4c0396d5f86e2cca376c in term 1.
I20260927 11:20:29.206230 13487 leader_election.cc:302] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be8abbc2c5fa4c0396d5f86e2cca376c, da78c16a4f4247eabd2d9be3dd0ed162; no voters:
I20260927 11:20:29.206367 13406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed71ca5f2c3043ec8a2944aee2aa358a" candidate_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72c77f4f0cec454d96ef056c3b2ec1fd"
I20260927 11:20:29.206446 13406 raft_consensus.cc:3037] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:29.207325 13406 raft_consensus.cc:2454] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be8abbc2c5fa4c0396d5f86e2cca376c in term 1.
I20260927 11:20:29.207563 13640 raft_consensus.cc:2781] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:20:29.207731 13640 raft_consensus.cc:686] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [term 1 LEADER]: Becoming Leader. State: Replica: be8abbc2c5fa4c0396d5f86e2cca376c, State: Running, Role: LEADER
I20260927 11:20:29.207828 13640 consensus_queue.cc:237] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da78c16a4f4247eabd2d9be3dd0ed162" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46823 } } peers { permanent_uuid: "72c77f4f0cec454d96ef056c3b2ec1fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35271 } } peers { permanent_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } }
I20260927 11:20:29.208653 13133 catalog_manager.cc:5495] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c reported cstate change: term changed from 0 to 1, leader changed from <none> to be8abbc2c5fa4c0396d5f86e2cca376c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da78c16a4f4247eabd2d9be3dd0ed162" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72c77f4f0cec454d96ef056c3b2ec1fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be8abbc2c5fa4c0396d5f86e2cca376c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } health_report { overall_health: HEALTHY } } }
I20260927 11:20:29.238265 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:29.229367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.0.98.3:36163)
0927 11:20:29.229442 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:29.229444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:29.229452 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:20:29.229496 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:29.229497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:29.229522 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:29.229593 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:20:29.229822 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:29.230575 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:29.232586 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:29.237335 (+ 4749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:29.237426 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:29.237671 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:20:29.237822 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:20:29.237896 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:29.238141 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 11:20:29.238164 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 11:20:29.240454 13466 tablet.cc:2369] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:20:29.246253 13406 raft_consensus.cc:1261] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd [term 1 FOLLOWER]: Refusing update from remote peer be8abbc2c5fa4c0396d5f86e2cca376c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:20:29.246253 13285 raft_consensus.cc:1261] T ed71ca5f2c3043ec8a2944aee2aa358a P da78c16a4f4247eabd2d9be3dd0ed162 [term 1 FOLLOWER]: Refusing update from remote peer be8abbc2c5fa4c0396d5f86e2cca376c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:20:29.246618 13640 consensus_queue.cc:1035] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [LEADER]: Connected to new peer: Peer: permanent_uuid: "72c77f4f0cec454d96ef056c3b2ec1fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:20:29.246758 13645 consensus_queue.cc:1035] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da78c16a4f4247eabd2d9be3dd0ed162" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:20:29.250679 13648 mvcc.cc:205] Tried to move back new op lower bound from 7333920887789338624 to 7333920887635771392. Current Snapshot: MvccSnapshot[applied={T|T < 7333920887789338624}]
I20260927 11:20:29.251086 13650 mvcc.cc:205] Tried to move back new op lower bound from 7333920887789338624 to 7333920887635771392. Current Snapshot: MvccSnapshot[applied={T|T < 7333920887789338624}]
I20260927 11:20:29.253011 13132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:20:29.253118 13132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:20:29.253744 13649 mvcc.cc:205] Tried to move back new op lower bound from 7333920887789338624 to 7333920887635771392. Current Snapshot: MvccSnapshot[applied={T|T < 7333920887789338624}]
I20260927 11:20:29.253912 13132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9a6693c40954f83b2c272401c22dd5d: Sending DeleteTablet for 3 replicas of tablet ed71ca5f2c3043ec8a2944aee2aa358a
I20260927 11:20:29.254544 13265 tablet_service.cc:1514] Processing DeleteTablet for tablet ed71ca5f2c3043ec8a2944aee2aa358a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45258
I20260927 11:20:29.254799 13399 tablet_service.cc:1514] Processing DeleteTablet for tablet ed71ca5f2c3043ec8a2944aee2aa358a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52262
I20260927 11:20:29.254992 13533 tablet_service.cc:1514] Processing DeleteTablet for tablet ed71ca5f2c3043ec8a2944aee2aa358a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40552
I20260927 11:20:29.255448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 13171
I20260927 11:20:29.257493 13658 tablet_replica.cc:330] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd: stopping tablet replica
I20260927 11:20:29.257622 13658 raft_consensus.cc:2229] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:20:29.257725 13658 raft_consensus.cc:2258] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:20:29.257884 13659 tablet_replica.cc:330] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c: stopping tablet replica
I20260927 11:20:29.257978 13659 raft_consensus.cc:2229] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:20:29.258102 13659 raft_consensus.cc:2258] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:20:29.258577 13659 ts_tablet_manager.cc:1905] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:20:29.260454 13659 ts_tablet_manager.cc:1918] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:20:29.260535 13659 log.cc:1196] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/ed71ca5f2c3043ec8a2944aee2aa358a
I20260927 11:20:29.260840 13659 ts_tablet_manager.cc:1939] T ed71ca5f2c3043ec8a2944aee2aa358a P be8abbc2c5fa4c0396d5f86e2cca376c: Deleting consensus metadata
I20260927 11:20:29.262470 13117 catalog_manager.cc:4841] TS be8abbc2c5fa4c0396d5f86e2cca376c (127.0.98.3:36163): tablet ed71ca5f2c3043ec8a2944aee2aa358a (table test-table [id=134fc5eae3ea44bcbc588659c92d3c44]) successfully deleted
I20260927 11:20:29.270591 13658 ts_tablet_manager.cc:1905] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 11:20:29.270663 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46823 (error 108)
I20260927 11:20:29.272635 13658 ts_tablet_manager.cc:1918] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:20:29.272732 13658 log.cc:1196] T ed71ca5f2c3043ec8a2944aee2aa358a P 72c77f4f0cec454d96ef056c3b2ec1fd: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/ed71ca5f2c3043ec8a2944aee2aa358a
I20260927 11:20:29.272979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 13335
I20260927 11:20:29.280814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 13468
I20260927 11:20:29.281811 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:29.281362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35271 (local address 127.0.0.1:52266)
0927 11:20:29.281713 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:29.281764 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315}
W20260927 11:20:29.281958 13120 catalog_manager.cc:4584] TS 72c77f4f0cec454d96ef056c3b2ec1fd (127.0.98.2:35271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed71ca5f2c3043ec8a2944aee2aa358a: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35271: connect: Connection refused (error 111)
I20260927 11:20:29.282989 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:29.281978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46823 (local address 127.0.0.1:45266)
0927 11:20:29.282926 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:29.282956 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":929}
I20260927 11:20:29.286237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 13100
I20260927 11:20:29.292750 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 11:20:29.637580 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 11:20:29.637707 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 11:20:29.982498 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 11:20:29.982 UTC [12167] LOG: received smart shutdown request
2026-09-27 11:20:29.986 UTC [12167] LOG: background worker "logical replication launcher" (PID 12176) exited with exit code 1
2026-09-27 11:20:29.986 UTC [12171] LOG: shutting down
2026-09-27 11:20:30.155 UTC [12167] LOG: database system is shut down
2026-09-27T11:20:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47609 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:20:30 dist-test-slave-lsvg krb5kdc[13669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:20:30 dist-test-slave-lsvg krb5kdc[13669](info): set up 2 sockets
Sep 27 11:20:30 dist-test-slave-lsvg krb5kdc[13669](info): commencing operation
krb5kdc: starting...
W20260927 11:20:31.316963 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:20:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:20:31.435491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41247
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41247,127.0.98.61:42763,127.0.98.60:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:31.520608 13682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:31.520879 13682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:31.520906 13682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:31.522213 13682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:20:31.522260 13682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:31.522277 13682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:31.522291 13682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:20:31.522305 13682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:20:31.523979 13682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41247,127.0.98.61:42763,127.0.98.60:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:31.524252 13682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:31.524501 13682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:31.529450 13688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:31.529549 13682 server_base.cc:1034] running on GCE node
W20260927 11:20:31.529479 13687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:31.529456 13690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:31.530077 13682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:31.530390 13682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:31.531590 13682 hybrid_clock.cc:648] HybridClock initialized: now 1790508031531582 us; error 64 us; skew 500 ppm
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:31.539050 13682 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:20:31.539482 13682 webserver.cc:460] Webserver started at http://127.0.98.62:35639/ using document root <none> and password file <none>
I20260927 11:20:31.539739 13682 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:31.539844 13682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:31.540022 13682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:20:31.541155 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "69e419f083dd4507a4b90fae40b00295"
format_stamp: "Formatted at 2026-09-27 11:20:31 on dist-test-slave-lsvg"
server_key: "ee9be5ee6c56bc6211802a04bb6f4a69"
server_key_iv: "c2e0b345c85dfe0f3725f24ef893f525"
server_key_version: "encryptionkey@0"
I20260927 11:20:31.541529 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "69e419f083dd4507a4b90fae40b00295"
format_stamp: "Formatted at 2026-09-27 11:20:31 on dist-test-slave-lsvg"
server_key: "ee9be5ee6c56bc6211802a04bb6f4a69"
server_key_iv: "c2e0b345c85dfe0f3725f24ef893f525"
server_key_version: "encryptionkey@0"
I20260927 11:20:31.542932 13682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:20:31.543732 13697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:31.543915 13682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:20:31.544026 13682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "69e419f083dd4507a4b90fae40b00295"
format_stamp: "Formatted at 2026-09-27 11:20:31 on dist-test-slave-lsvg"
server_key: "ee9be5ee6c56bc6211802a04bb6f4a69"
server_key_iv: "c2e0b345c85dfe0f3725f24ef893f525"
server_key_version: "encryptionkey@0"
I20260927 11:20:31.544113 13682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:31.557893 13682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:31.560509 13682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:31.560680 13682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:31.564607 13682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41247
I20260927 11:20:31.564636 13749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41247 every 8 connection(s)
I20260927 11:20:31.564994 13682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:20:31.565914 13750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:31.567924 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:20:31.570585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 13682
I20260927 11:20:31.570667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:20:31.570861 392 external_mini_cluster.cc:1428] Setting key c4b1cfc4467c96483baa002e91456043
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:31.578943 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.568639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41247 (local address 127.0.0.1:58240)
0927 11:20:31.568791 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.568832 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:31.569328 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:31.569594 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:31.569596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:31.569884 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:31.570041 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.570045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.571151 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.571153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.572295 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.572298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.573040 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.573041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.573163 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.573702 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:31.573710 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:31.575107 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:31.577426 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.577433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.577435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.577634 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.577908 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.577908 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.577914 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.578000 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.578467 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:31.578472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:31.578527 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 11:20:31.578596 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 11:20:31.578723 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260927 11:20:31.578949 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.568795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.0.98.62:41247)
0927 11:20:31.568914 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.568918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.568954 (+ 36us) server_negotiation.cc:407] Connection header received
0927 11:20:31.569463 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.569464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.569491 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.569569 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.570121 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.571067 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.572396 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.572940 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.573095 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.575198 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.575206 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.575207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.575215 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.577291 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.577724 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.577728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.577729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.577830 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.578072 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.578073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.578074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.578386 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.578493 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.578662 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.578810 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260927 11:20:31.579806 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:20:31.580165 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.580035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42763 (local address 127.0.0.1:48182)
0927 11:20:31.580091 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.580144 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 11:20:31.580348 13701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42763: connect: Connection refused (error 111)
I20260927 11:20:31.580756 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.580678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42763 (local address 127.0.0.1:48194)
0927 11:20:31.580717 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.580741 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 11:20:31.580942 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42763: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42763: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 11:20:31.596868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42763
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41247,127.0.98.61:42763,127.0.98.60:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:20:31.630158 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } attempt: 1
I20260927 11:20:31.630659 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.630527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42763 (local address 127.0.0.1:48198)
0927 11:20:31.630588 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.630631 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 11:20:31.631013 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.630917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42763 (local address 127.0.0.1:48200)
0927 11:20:31.630960 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.630987 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 11:20:31.631105 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42763: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42763: connect: Connection refused (error 111)
W20260927 11:20:31.679929 13757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:31.680137 13757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:31.680160 13757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:31.681850 13757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:20:31.681895 13757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:31.681910 13757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:31.681922 13757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:20:31.681973 13757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:20:31.683487 13757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41247,127.0.98.61:42763,127.0.98.60:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:31.683743 13757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:31.683965 13757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:20:31.688978 13757 server_base.cc:1034] running on GCE node
W20260927 11:20:31.689136 13765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:31.689152 13762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:31.689172 13763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:31.689584 13757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:31.689934 13757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:31.691082 13757 hybrid_clock.cc:648] HybridClock initialized: now 1790508031691063 us; error 38 us; skew 500 ppm
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:31.693769 13757 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 11:20:31.694223 13757 webserver.cc:460] Webserver started at http://127.0.98.61:40557/ using document root <none> and password file <none>
I20260927 11:20:31.694463 13757 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:31.694528 13757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:31.694650 13757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:20:31.695719 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/data/instance:
uuid: "b0f9aa0cba564a9da82b224922ca619b"
format_stamp: "Formatted at 2026-09-27 11:20:31 on dist-test-slave-lsvg"
server_key: "200937ae3a6413723d5c3428c6da7c73"
server_key_iv: "8e923f14f6ccfae1acc65f7085981822"
server_key_version: "encryptionkey@0"
I20260927 11:20:31.696075 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/wal/instance:
uuid: "b0f9aa0cba564a9da82b224922ca619b"
format_stamp: "Formatted at 2026-09-27 11:20:31 on dist-test-slave-lsvg"
server_key: "200937ae3a6413723d5c3428c6da7c73"
server_key_iv: "8e923f14f6ccfae1acc65f7085981822"
server_key_version: "encryptionkey@0"
I20260927 11:20:31.697438 13757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:20:31.698179 13772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:31.698369 13757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:20:31.698472 13757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/wal
uuid: "b0f9aa0cba564a9da82b224922ca619b"
format_stamp: "Formatted at 2026-09-27 11:20:31 on dist-test-slave-lsvg"
server_key: "200937ae3a6413723d5c3428c6da7c73"
server_key_iv: "8e923f14f6ccfae1acc65f7085981822"
server_key_version: "encryptionkey@0"
I20260927 11:20:31.698553 13757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:31.699250 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } attempt: 2
I20260927 11:20:31.699636 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.699511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42763 (local address 127.0.0.1:48208)
0927 11:20:31.699569 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.699601 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 11:20:31.699934 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.699848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42763 (local address 127.0.0.1:48224)
0927 11:20:31.699879 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.699901 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260927 11:20:31.700029 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42763: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42763: connect: Connection refused (error 111)
I20260927 11:20:31.713322 13757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:31.715302 13757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:31.715474 13757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:31.719568 13757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42763
I20260927 11:20:31.719620 13824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42763 every 8 connection(s)
I20260927 11:20:31.719991 13757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/data/info.pb
I20260927 11:20:31.720885 13825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:31.722044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 13757
I20260927 11:20:31.722144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-1/wal/instance
I20260927 11:20:31.722312 392 external_mini_cluster.cc:1428] Setting key 0a231d84104e395817761e02ecf05659
I20260927 11:20:31.722798 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 11:20:31.733214 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.723685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.0.98.62:41247)
0927 11:20:31.723736 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.723739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.724010 (+ 271us) server_negotiation.cc:407] Connection header received
0927 11:20:31.724530 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.724531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.724549 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.724625 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.725298 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.726099 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.727596 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.727789 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.727848 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.730217 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.730219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.730219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.730228 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.731580 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.732113 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.732115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.732115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.732180 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.732447 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.732448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.732449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.732593 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.732679 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.732996 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.733109 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 11:20:31.733460 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.723656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41247 (local address 127.0.0.1:58246)
0927 11:20:31.723834 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.723884 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:31.724426 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:31.724699 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:31.724702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:31.724953 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:31.725209 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.725214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.726230 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.726230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.727501 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.727504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.727882 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.727883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.728050 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.728558 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:31.728568 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:31.730116 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:31.731668 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.731711 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.731713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.732021 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.732257 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.732258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.732258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.732372 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.732698 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:31.732705 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:31.732846 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 11:20:31.732954 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 11:20:31.733193 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260927 11:20:31.734051 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:20:31.741765 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.734326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42763 (local address 127.0.0.1:48230)
0927 11:20:31.734379 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.734389 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:31.734440 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:31.734679 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:31.734680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:31.734831 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:31.734874 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.734876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.735742 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.735743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.736241 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.736242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.736929 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.736930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.737002 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.737457 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:31.737462 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:31.738688 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:31.740736 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.740737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.740737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.740901 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.741132 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.741132 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.741133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.741187 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.741532 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:31.741532 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:31.741567 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 11:20:31.741619 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 11:20:31.741697 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 11:20:31.741916 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.734371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.0.98.61:42763)
0927 11:20:31.734537 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.734541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.734561 (+ 20us) server_negotiation.cc:407] Connection header received
0927 11:20:31.734586 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.734588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.734599 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.734680 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.734957 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.735660 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.736337 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.736846 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.736922 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.738769 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.738778 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.738779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.738787 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.740643 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.740983 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.740987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.740987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.741068 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.741251 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.741252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.741252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.741456 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.741554 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.741658 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.741782 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260927 11:20:31.742571 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:20:31.742924 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.742814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36953 (local address 127.0.0.1:40952)
0927 11:20:31.742860 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.742905 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 11:20:31.743115 13775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
I20260927 11:20:31.743571 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.743486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36953 (local address 127.0.0.1:40964)
0927 11:20:31.743531 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.743555 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 11:20:31.743729 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36953: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 11:20:31.752002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36953
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41247,127.0.98.61:42763,127.0.98.60:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:20:31.791196 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } attempt: 3
I20260927 11:20:31.792987 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } attempt: 1
I20260927 11:20:31.793466 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.793320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36953 (local address 127.0.0.1:40966)
0927 11:20:31.793384 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.793430 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 11:20:31.793829 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.793730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36953 (local address 127.0.0.1:40978)
0927 11:20:31.793780 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.793808 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 11:20:31.793905 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36953: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:20:31.799019 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.791692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.0.98.61:42763)
0927 11:20:31.791738 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.791740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.791789 (+ 49us) server_negotiation.cc:407] Connection header received
0927 11:20:31.791817 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.791817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.791830 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.791886 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.792204 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.793011 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.793989 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.794211 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.794304 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.796213 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.796215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.796216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.796223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.797810 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.798217 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.798218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.798219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.798280 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.798481 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.798482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.798483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.798610 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.798681 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.798846 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.798930 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:20:31.799068 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.791604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42763 (local address 127.0.0.1:48234)
0927 11:20:31.791672 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.791685 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:31.791733 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:31.791900 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:31.791900 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:31.792061 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:31.792132 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.792134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.793141 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.793142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.793913 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.793916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.794305 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.794306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.794388 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.794873 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:31.794879 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:31.796116 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:31.797936 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.797937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.797938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.798127 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.798348 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.798349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.798350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.798415 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.798683 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:31.798685 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:31.798733 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 11:20:31.798819 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 11:20:31.798929 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 11:20:31.799576 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:20:31.799911 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.799813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36953 (local address 127.0.0.1:40984)
0927 11:20:31.799861 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.799888 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 11:20:31.800220 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.800129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36953 (local address 127.0.0.1:40994)
0927 11:20:31.800178 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.800201 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 11:20:31.800315 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36953: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
I20260927 11:20:31.831553 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } attempt: 1
I20260927 11:20:31.832062 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.831910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36953 (local address 127.0.0.1:40996)
0927 11:20:31.831984 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.832034 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 11:20:31.832429 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.832330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36953 (local address 127.0.0.1:41006)
0927 11:20:31.832374 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.832402 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 11:20:31.832530 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36953: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
W20260927 11:20:31.834596 13832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:31.834815 13832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:31.834867 13832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:31.836021 13832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:20:31.836082 13832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:31.836108 13832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:31.836151 13832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:20:31.836181 13832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:20:31.837773 13832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41247,127.0.98.61:42763,127.0.98.60:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:31.838047 13832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:31.838271 13832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:31.843037 13838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:31.843050 13840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:31.843051 13837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:31.843303 13832 server_base.cc:1034] running on GCE node
I20260927 11:20:31.843500 13832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:31.843732 13832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:31.844868 13832 hybrid_clock.cc:648] HybridClock initialized: now 1790508031844854 us; error 38 us; skew 500 ppm
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:31.847349 13832 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 11:20:31.847820 13832 webserver.cc:460] Webserver started at http://127.0.98.60:34343/ using document root <none> and password file <none>
I20260927 11:20:31.848052 13832 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:31.848122 13832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:31.848278 13832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:20:31.849434 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/data/instance:
uuid: "33a6377ec72b43e980bee7bd3db9c08e"
format_stamp: "Formatted at 2026-09-27 11:20:31 on dist-test-slave-lsvg"
server_key: "1fb51e079d0335ee8b9b2695a4250375"
server_key_iv: "2d41186351d4a1f8f605089e0c054d08"
server_key_version: "encryptionkey@0"
I20260927 11:20:31.849799 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/wal/instance:
uuid: "33a6377ec72b43e980bee7bd3db9c08e"
format_stamp: "Formatted at 2026-09-27 11:20:31 on dist-test-slave-lsvg"
server_key: "1fb51e079d0335ee8b9b2695a4250375"
server_key_iv: "2d41186351d4a1f8f605089e0c054d08"
server_key_version: "encryptionkey@0"
I20260927 11:20:31.851135 13832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:20:31.851933 13847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:31.852098 13832 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:20:31.852198 13832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/wal
uuid: "33a6377ec72b43e980bee7bd3db9c08e"
format_stamp: "Formatted at 2026-09-27 11:20:31 on dist-test-slave-lsvg"
server_key: "1fb51e079d0335ee8b9b2695a4250375"
server_key_iv: "2d41186351d4a1f8f605089e0c054d08"
server_key_version: "encryptionkey@0"
I20260927 11:20:31.852283 13832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:31.862039 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } attempt: 2
I20260927 11:20:31.862473 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.862342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36953 (local address 127.0.0.1:41010)
0927 11:20:31.862405 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.862445 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 11:20:31.862830 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.862736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36953 (local address 127.0.0.1:41012)
0927 11:20:31.862782 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.862806 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 11:20:31.862910 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36953: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36953: connect: Connection refused (error 111)
I20260927 11:20:31.872020 13832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:31.874112 13832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:31.874280 13832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:31.878433 13832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36953
I20260927 11:20:31.878472 13899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36953 every 8 connection(s)
I20260927 11:20:31.878865 13832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/data/info.pb
I20260927 11:20:31.879756 13900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:31.881371 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:20:31.887099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 13832
I20260927 11:20:31.887204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507878147783-392-0/minicluster-data/master-2/wal/instance
I20260927 11:20:31.887389 392 external_mini_cluster.cc:1428] Setting key 359f342db7291fc4a1b10cbf8e0f295f
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:31.891569 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.882159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.0.98.62:41247)
0927 11:20:31.882235 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.882238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.882432 (+ 194us) server_negotiation.cc:407] Connection header received
0927 11:20:31.882948 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.882949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.882963 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.883035 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.883679 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.884162 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.885245 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.885447 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.885516 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.888039 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.888040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.888041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.888049 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.889794 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.890306 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.890307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.890308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.890401 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.890762 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.890764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.890764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.890912 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.890982 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.891386 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.891474 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 11:20:31.891783 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.882138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41247 (local address 127.0.0.1:58256)
0927 11:20:31.882310 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.882346 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:31.882832 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:31.883131 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:31.883134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:31.883383 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:31.883583 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.883586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.884260 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.884261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.885131 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.885134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.885537 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.885538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.885639 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.886184 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:31.886192 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:31.887929 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:31.889880 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.889885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.889887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.890195 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.890495 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.890509 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.890509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.890682 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.890993 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:31.890999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:31.891206 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 11:20:31.891355 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 11:20:31.891525 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":4,"thread_start_us":72,"threads_started":1}
I20260927 11:20:31.892374 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:20:31.897091 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.887807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.0.98.60:36953)
0927 11:20:31.887964 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.887968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.888043 (+ 75us) server_negotiation.cc:407] Connection header received
0927 11:20:31.888124 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.888127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.888141 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.888228 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.888805 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.889657 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.890635 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.891149 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.891217 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.894103 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.894109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.894109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.894116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.895782 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.896198 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.896201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.896201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.896261 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.896478 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.896479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.896480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.896617 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.896773 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.896846 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.896925 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260927 11:20:31.899232 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.892644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42763 (local address 127.0.0.1:48250)
0927 11:20:31.892694 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.892709 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:31.892744 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:31.892939 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:31.892940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:31.893145 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:31.893210 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.893212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.893927 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.893928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.894496 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.894498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.894886 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.894887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.894960 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.895316 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:31.895320 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:31.896414 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:31.898286 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.898287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.898288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.898509 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.898707 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.898708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.898708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.898759 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.899021 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:31.899022 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:31.899054 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 11:20:31.899104 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 11:20:31.899168 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 11:20:31.899292 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.892744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.0.98.61:42763)
0927 11:20:31.892821 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.892823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.892831 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:20:31.892855 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.892856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.892866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.892936 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.893297 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.893825 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.894596 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.894806 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.894878 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.896504 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.896506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.896506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.896514 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.898209 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.898578 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.898579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.898579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.898640 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.898819 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.898820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.898820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.898955 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.899022 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.899128 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.899202 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 11:20:31.899612 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:20:31.906431 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.897862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.0.98.62:41247)
0927 11:20:31.897910 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.897912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.897920 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:20:31.897942 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.897942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.897951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.898020 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.898573 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.899333 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.900327 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.900611 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.900688 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.903233 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.903235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.903235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.903244 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.904920 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.905355 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.905356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.905356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.905408 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.905616 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.905616 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.905616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.905713 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.905896 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.906208 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.906295 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 11:20:31.906795 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.899862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36953 (local address 127.0.0.1:41042)
0927 11:20:31.899925 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.899935 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:31.899972 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:31.900082 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:31.900083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:31.900220 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:31.900251 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.900253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.901018 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.901019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.901642 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.901645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.902158 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.902159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.902208 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.902654 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:31.902658 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:31.904055 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:31.905784 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.905785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.905785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.905940 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.906140 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.906141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.906141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.906226 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.906494 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:31.906495 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:31.906543 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 11:20:31.906614 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:20:31.906700 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 11:20:31.907079 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.899894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.0.98.60:36953)
0927 11:20:31.899922 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.899924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.899960 (+ 36us) server_negotiation.cc:407] Connection header received
0927 11:20:31.900022 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.900023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.900031 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.900072 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.900322 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.900947 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.901733 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.902079 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.902148 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.904153 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.904154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.904154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.904161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.905713 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.906014 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.906015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.906016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.906068 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.906303 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.906304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.906305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.906429 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.906503 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.906913 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.906983 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260927 11:20:31.907688 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } attempt: 2
I20260927 11:20:31.908571 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e: Bootstrap starting.
I20260927 11:20:31.909672 13900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:31.910028 13900 log.cc:824] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:31.910989 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e: No bootstrap required, opened a new log
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:20:31.913225 13900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } }
I20260927 11:20:31.913394 13900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:31.913463 13900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a6377ec72b43e980bee7bd3db9c08e, State: Initialized, Role: FOLLOWER
I20260927 11:20:31.913605 13900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } }
I20260927 11:20:31.913880 13906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } } }
I20260927 11:20:31.913944 13900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:20:31.913959 13906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:20:31.915596 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.907083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.0.98.61:42763)
0927 11:20:31.907125 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.907128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.907136 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:20:31.907192 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.907194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.907207 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.907240 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.907874 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.908322 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.909265 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.909457 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.909511 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.912091 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.912093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.912093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.912100 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.914312 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.914802 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.914803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.914803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.914863 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.915054 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.915055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.915055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.915187 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.915363 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.915410 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.915483 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 11:20:31.916157 13917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:20:31.916208 13917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:20:31.916971 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:20:31.917829 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.910435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36953 (local address 127.0.0.1:41058)
0927 11:20:31.910488 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.910511 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:31.910552 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:31.910704 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:31.910705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:31.910841 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:31.910898 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.910900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.911592 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.911593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.912170 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.912172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.912643 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.912645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.912711 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.913343 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:31.913348 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:31.914766 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:31.916716 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.916717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.916717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.916936 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.917161 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.917162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.917163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.917223 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.917504 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:31.917505 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:31.917552 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 11:20:31.917634 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 11:20:31.917732 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 11:20:31.917837 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.910487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41058 (local address 127.0.98.60:36953)
0927 11:20:31.910523 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.910525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.910537 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:20:31.910617 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.910618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.910625 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.910697 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.910983 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.911500 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.912256 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.912531 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.912634 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.914861 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.914862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.914862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.914868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.916610 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.917016 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.917017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.917017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.917097 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.917297 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.917298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.917298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.917428 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.917509 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.917671 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.917742 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:20:31.916244 13832 master_runner.cc:186] Error getting master registration for 127.0.98.62:41247: OK, instance_id { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" instance_seqno: 1790508031563522 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 11:20:31.919492 13832 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 11:20:31.920229 13750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295: Bootstrap starting.
I20260927 11:20:31.920790 13750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:31.921541 13750 log.cc:824] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:31.922379 13750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295: No bootstrap required, opened a new log
I20260927 11:20:31.926610 13750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } }
I20260927 11:20:31.926836 13750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:31.926934 13750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e419f083dd4507a4b90fae40b00295, State: Initialized, Role: FOLLOWER
I20260927 11:20:31.927106 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.918577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.0.98.62:41247)
0927 11:20:31.918617 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.918619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.919186 (+ 567us) server_negotiation.cc:407] Connection header received
0927 11:20:31.919250 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.919251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.919263 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.919319 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.920077 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.920764 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.921891 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.922187 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.922237 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.923862 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.923863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.923864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.923871 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.925676 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.926114 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.926116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.926116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.926174 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.926403 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.926404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.926405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.926544 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.926647 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.926812 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.926894 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:20:31.927670 13750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } }
I20260927 11:20:31.928118 13750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:20:31.929186 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.921141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.0.98.61:42763)
0927 11:20:31.921185 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.921188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.921195 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:20:31.921217 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.921218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.921227 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.921290 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.922005 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.922694 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.923719 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.924022 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.924083 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.925490 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.925492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.925492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.925499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.927330 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.928076 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.928077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.928077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.928137 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.928323 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.928324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.928324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.928451 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.928536 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.928971 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.929090 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 11:20:31.931113 13938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:20:31.931228 13938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:20:31.931517 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } } }
I20260927 11:20:31.931658 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:20:31.930778 13682 master_runner.cc:186] Error getting master registration for 127.0.98.61:42763: OK, instance_id { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" instance_seqno: 1790508031718403 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 11:20:31.933799 13682 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 11:20:31.934674 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.918902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.0.98.60:36953)
0927 11:20:31.918940 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.918942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.927165 (+ 8223us) server_negotiation.cc:407] Connection header received
0927 11:20:31.927275 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.927277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.927290 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.927323 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.927884 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.928577 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.929678 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.929972 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.930013 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.931573 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.931575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.931575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.931583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.933365 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.933768 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.933770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.933770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.933827 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.934064 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.934066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.934066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.934216 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.934330 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.934487 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.934577 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:20:31.954052 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } attempt: 3
Sep 27 11:20:31 dist-test-slave-lsvg krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:20:31.962999 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.954328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36953 (local address 127.0.0.1:41074)
0927 11:20:31.954380 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:31.954391 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:31.954428 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:31.954601 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:31.954602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:31.954816 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:31.954883 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.954885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.955856 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.955857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.956679 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:31.956682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.957647 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.957648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:31.957713 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.958259 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:31.958265 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:31.959857 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:31.961829 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.961831 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.961831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.962063 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.962282 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:31.962283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:31.962283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:31.962353 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:31.962638 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:31.962639 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:31.962710 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 11:20:31.962780 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 11:20:31.962891 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 11:20:31.963790 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:31.954403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.0.98.60:36953)
0927 11:20:31.954463 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:31.954465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:31.954473 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:20:31.954496 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:31.954497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:31.954506 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:31.954583 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:31.954959 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.955766 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.956770 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:31.957571 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:31.957638 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:31.959955 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:31.959957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:31.959957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:31.959964 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:31.961745 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.962142 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.962143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.962144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.962208 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:31.962420 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:31.962421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:31.962421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:31.962567 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:31.962637 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:31.963613 (+ 976us) server_negotiation.cc:299] Negotiation successful
0927 11:20:31.963695 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 11:20:31.965607 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b: Bootstrap starting.
I20260927 11:20:31.966357 13825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:31.966691 13825 log.cc:824] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:31.967564 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b: No bootstrap required, opened a new log
I20260927 11:20:31.969631 13825 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } }
I20260927 11:20:31.969784 13825 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:31.969817 13825 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f9aa0cba564a9da82b224922ca619b, State: Initialized, Role: FOLLOWER
I20260927 11:20:31.969954 13825 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } }
I20260927 11:20:31.970281 13825 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:20:31.970229 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } } }
I20260927 11:20:31.970418 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [sys.catalog]: This master's current role is: FOLLOWER
W20260927 11:20:31.972186 13952 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:20:31.972290 13952 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:20:31.974370 13757 master_runner.cc:263] No leader master found from master b0f9aa0cba564a9da82b224922ca619b
I20260927 11:20:31.974428 13757 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 11:20:32.021279 13927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:20:32.021436 13927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } }
I20260927 11:20:32.021729 13927 leader_election.cc:288] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0f9aa0cba564a9da82b224922ca619b (127.0.98.61:42763), 33a6377ec72b43e980bee7bd3db9c08e (127.0.98.60:36953)
I20260927 11:20:32.021929 13941 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:20:32.022042 13941 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } }
I20260927 11:20:32.022272 13941 leader_election.cc:288] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69e419f083dd4507a4b90fae40b00295 (127.0.98.62:41247), 33a6377ec72b43e980bee7bd3db9c08e (127.0.98.60:36953)
I20260927 11:20:32.022680 13799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69e419f083dd4507a4b90fae40b00295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f9aa0cba564a9da82b224922ca619b" is_pre_election: true
I20260927 11:20:32.022814 13799 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69e419f083dd4507a4b90fae40b00295 in term 0.
I20260927 11:20:32.022826 13874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b0f9aa0cba564a9da82b224922ca619b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a6377ec72b43e980bee7bd3db9c08e" is_pre_election: true
I20260927 11:20:32.023003 13874 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f9aa0cba564a9da82b224922ca619b in term 0.
I20260927 11:20:32.023187 13701 leader_election.cc:302] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69e419f083dd4507a4b90fae40b00295, b0f9aa0cba564a9da82b224922ca619b; no voters:
I20260927 11:20:32.023428 13724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b0f9aa0cba564a9da82b224922ca619b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e419f083dd4507a4b90fae40b00295" is_pre_election: true
I20260927 11:20:32.023540 13724 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f9aa0cba564a9da82b224922ca619b in term 0.
I20260927 11:20:32.023674 13927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:20:32.023727 13927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:20:32.023756 13927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:32.024698 13927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } }
I20260927 11:20:32.024834 13927 leader_election.cc:288] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [CANDIDATE]: Term 1 election: Requested vote from peers b0f9aa0cba564a9da82b224922ca619b (127.0.98.61:42763), 33a6377ec72b43e980bee7bd3db9c08e (127.0.98.60:36953)
I20260927 11:20:32.025086 13799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69e419f083dd4507a4b90fae40b00295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f9aa0cba564a9da82b224922ca619b"
I20260927 11:20:32.025178 13799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:32.026089 13799 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69e419f083dd4507a4b90fae40b00295 in term 1.
I20260927 11:20:32.026317 13701 leader_election.cc:302] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69e419f083dd4507a4b90fae40b00295, b0f9aa0cba564a9da82b224922ca619b; no voters:
I20260927 11:20:32.026432 13775 leader_election.cc:302] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69e419f083dd4507a4b90fae40b00295, b0f9aa0cba564a9da82b224922ca619b; no voters:
I20260927 11:20:32.026602 13941 raft_consensus.cc:2741] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 11:20:32.026844 13873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69e419f083dd4507a4b90fae40b00295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a6377ec72b43e980bee7bd3db9c08e"
I20260927 11:20:32.026950 13873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:32.027954 13873 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69e419f083dd4507a4b90fae40b00295 in term 1.
I20260927 11:20:32.028065 13874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69e419f083dd4507a4b90fae40b00295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a6377ec72b43e980bee7bd3db9c08e" is_pre_election: true
I20260927 11:20:32.028139 13874 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 69e419f083dd4507a4b90fae40b00295 in term 1. Re-sending same reply.
I20260927 11:20:32.028394 13927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:20:32.028568 13927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [term 1 LEADER]: Becoming Leader. State: Replica: 69e419f083dd4507a4b90fae40b00295, State: Running, Role: LEADER
I20260927 11:20:32.028666 13927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } }
I20260927 11:20:32.029100 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e419f083dd4507a4b90fae40b00295. Latest consensus state: current_term: 1 leader_uuid: "69e419f083dd4507a4b90fae40b00295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } } }
I20260927 11:20:32.029203 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:32.030866 13956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:20:32.031078 13956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:20:32.032727 13874 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [term 1 FOLLOWER]: Refusing update from remote peer 69e419f083dd4507a4b90fae40b00295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:20:32.033185 13954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:20:32.034777 13906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e419f083dd4507a4b90fae40b00295. Latest consensus state: current_term: 1 leader_uuid: "69e419f083dd4507a4b90fae40b00295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } } }
I20260927 11:20:32.034894 13906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:20:32.035288 13799 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [term 1 FOLLOWER]: Refusing update from remote peer 69e419f083dd4507a4b90fae40b00295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:20:32.035540 13954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:20:32.036760 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e419f083dd4507a4b90fae40b00295. Latest consensus state: current_term: 1 leader_uuid: "69e419f083dd4507a4b90fae40b00295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } } }
I20260927 11:20:32.036857 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:20:32.038130 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69e419f083dd4507a4b90fae40b00295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } } }
I20260927 11:20:32.038231 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:32.038676 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69e419f083dd4507a4b90fae40b00295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } } }
I20260927 11:20:32.038764 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:20:32.038961 13957 mvcc.cc:205] Tried to move back new op lower bound from 7333920899203252224 to 7333920899189878784. Current Snapshot: MvccSnapshot[applied={T|T < 7333920899203252224}]
I20260927 11:20:32.039130 13960 mvcc.cc:205] Tried to move back new op lower bound from 7333920899203252224 to 7333920899189878784. Current Snapshot: MvccSnapshot[applied={T|T < 7333920899203252224}]
I20260927 11:20:32.039934 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69e419f083dd4507a4b90fae40b00295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } } }
I20260927 11:20:32.040021 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295 [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:32.040335 13906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69e419f083dd4507a4b90fae40b00295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e419f083dd4507a4b90fae40b00295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41247 } } peers { permanent_uuid: "b0f9aa0cba564a9da82b224922ca619b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42763 } } peers { permanent_uuid: "33a6377ec72b43e980bee7bd3db9c08e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36953 } } }
I20260927 11:20:32.040424 13906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:20:32.041584 13956 catalog_manager.cc:1347] Generated new cluster ID: 457ccbf82d19476583639011904b52be
I20260927 11:20:32.041641 13956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:20:32.055141 13958 mvcc.cc:205] Tried to move back new op lower bound from 7333920899203252224 to 7333920899189878784. Current Snapshot: MvccSnapshot[applied={T|T < 7333920899203252224}]
I20260927 11:20:32.067195 13956 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:20:32.067938 13956 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:20:32.080921 13956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69e419f083dd4507a4b90fae40b00295: Generated new TSK 0
I20260927 11:20:32.081183 13956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:20:32.916680 13917 catalog_manager.cc:1259] Loaded cluster ID: 457ccbf82d19476583639011904b52be
I20260927 11:20:32.916776 13917 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e: loading cluster ID for follower catalog manager: success
I20260927 11:20:32.917665 13917 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e: acquiring CA information for follower catalog manager: success
I20260927 11:20:32.917937 13917 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 33a6377ec72b43e980bee7bd3db9c08e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 11:20:32.972744 13952 catalog_manager.cc:1259] Loaded cluster ID: 457ccbf82d19476583639011904b52be
I20260927 11:20:32.972820 13952 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b: loading cluster ID for follower catalog manager: success
I20260927 11:20:32.973635 13952 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b: acquiring CA information for follower catalog manager: success
I20260927 11:20:32.973901 13952 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b0f9aa0cba564a9da82b224922ca619b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 11:20:33.130097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 13682
I20260927 11:20:33.135378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 13757
I20260927 11:20:33.140408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 13832
2026-09-27T11:20:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2870 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:20:33 dist-test-slave-lsvg krb5kdc[13971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:20:33 dist-test-slave-lsvg krb5kdc[13971](info): set up 2 sockets
Sep 27 11:20:33 dist-test-slave-lsvg krb5kdc[13971](info): commencing operation
krb5kdc: starting...
W20260927 11:20:34.181211 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:20:34 dist-test-slave-lsvg krb5kdc[13971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:20:34.302304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33325
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:34.386423 13984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:34.386631 13984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:34.386653 13984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:34.387792 13984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:20:34.387832 13984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:34.387847 13984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:34.387859 13984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:20:34.387871 13984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:20:34.389402 13984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:34.389639 13984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:34.389858 13984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:34.394528 13992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:34.394621 13990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:34.394554 13989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:34.394639 13984 server_base.cc:1034] running on GCE node
I20260927 11:20:34.395020 13984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:34.395303 13984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:34.396466 13984 hybrid_clock.cc:648] HybridClock initialized: now 1790508034396435 us; error 53 us; skew 500 ppm
Sep 27 11:20:34 dist-test-slave-lsvg krb5kdc[13971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:34.398916 13984 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:20:34.399380 13984 webserver.cc:460] Webserver started at http://127.0.98.62:35119/ using document root <none> and password file <none>
I20260927 11:20:34.399602 13984 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:34.399672 13984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:34.399821 13984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:20:34.400868 13984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "38cdc2fc323c4d7283177941c42aa7ab"
format_stamp: "Formatted at 2026-09-27 11:20:34 on dist-test-slave-lsvg"
server_key: "282092ca038fc72f76a326400bdfee4f"
server_key_iv: "3c63e00c313ddbf9f3d3cc9f03670641"
server_key_version: "encryptionkey@0"
I20260927 11:20:34.401247 13984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "38cdc2fc323c4d7283177941c42aa7ab"
format_stamp: "Formatted at 2026-09-27 11:20:34 on dist-test-slave-lsvg"
server_key: "282092ca038fc72f76a326400bdfee4f"
server_key_iv: "3c63e00c313ddbf9f3d3cc9f03670641"
server_key_version: "encryptionkey@0"
I20260927 11:20:34.402606 13984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:20:34.403347 13999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:34.403537 13984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:20:34.403642 13984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "38cdc2fc323c4d7283177941c42aa7ab"
format_stamp: "Formatted at 2026-09-27 11:20:34 on dist-test-slave-lsvg"
server_key: "282092ca038fc72f76a326400bdfee4f"
server_key_iv: "3c63e00c313ddbf9f3d3cc9f03670641"
server_key_version: "encryptionkey@0"
I20260927 11:20:34.403714 13984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:34.415181 13984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:34.417235 13984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:34.417407 13984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:34.421435 13984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33325
I20260927 11:20:34.421490 14051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33325 every 8 connection(s)
I20260927 11:20:34.421867 13984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:20:34.422705 14052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:34.424989 14052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab: Bootstrap starting.
I20260927 11:20:34.425623 14052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:34.425917 14052 log.cc:824] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:34.426630 14052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab: No bootstrap required, opened a new log
I20260927 11:20:34.427414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 13984
I20260927 11:20:34.427518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:20:34.427726 392 external_mini_cluster.cc:1428] Setting key 020ab8e029a5ed055c890c6a21f5c465
I20260927 11:20:34.428797 14052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38cdc2fc323c4d7283177941c42aa7ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33325 } }
I20260927 11:20:34.428970 14052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:34.429009 14052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cdc2fc323c4d7283177941c42aa7ab, State: Initialized, Role: FOLLOWER
I20260927 11:20:34.429189 14052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38cdc2fc323c4d7283177941c42aa7ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33325 } }
I20260927 11:20:34.429288 14052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:20:34.429322 14052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:20:34.429376 14052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:34.430368 14052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38cdc2fc323c4d7283177941c42aa7ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33325 } }
I20260927 11:20:34.430456 14052 leader_election.cc:302] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38cdc2fc323c4d7283177941c42aa7ab; no voters:
I20260927 11:20:34.430634 14052 leader_election.cc:288] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:20:34.430675 14057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:20:34.430825 14057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [term 1 LEADER]: Becoming Leader. State: Replica: 38cdc2fc323c4d7283177941c42aa7ab, State: Running, Role: LEADER
I20260927 11:20:34.430895 14052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:20:34.430953 14057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38cdc2fc323c4d7283177941c42aa7ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33325 } }
I20260927 11:20:34.431277 14057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38cdc2fc323c4d7283177941c42aa7ab. Latest consensus state: current_term: 1 leader_uuid: "38cdc2fc323c4d7283177941c42aa7ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38cdc2fc323c4d7283177941c42aa7ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33325 } } }
I20260927 11:20:34.431370 14057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:34.431291 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38cdc2fc323c4d7283177941c42aa7ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38cdc2fc323c4d7283177941c42aa7ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33325 } } }
I20260927 11:20:34.431442 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:34.431742 14062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:20:34.432294 14062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:20:34.434019 14062 catalog_manager.cc:1347] Generated new cluster ID: 9529a4dc9cbd4bb590acc4a507d9c10d
I20260927 11:20:34.434078 14062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:20:34 dist-test-slave-lsvg krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:34.439329 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:34.428244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.0.98.62:33325)
0927 11:20:34.428434 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:34.428443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:34.428468 (+ 25us) server_negotiation.cc:407] Connection header received
0927 11:20:34.428778 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:34.428784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:34.428830 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:34.428941 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:34.429811 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:34.430988 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:34.432142 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:34.432819 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:34.432932 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:34.435757 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:34.435761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:34.435761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:34.435773 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:34.437710 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:34.438114 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:34.438117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:34.438117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:34.438236 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:34.438453 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:34.438453 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:34.438454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:34.438741 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:34.438895 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:34.439001 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 11:20:34.439130 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260927 11:20:34.440418 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:20:34.454963 14062 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:20:34.455530 14062 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:20:34.460937 14062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38cdc2fc323c4d7283177941c42aa7ab: Generated new TSK 0
I20260927 11:20:34.461182 14062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:20:34.494645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 13984
2026-09-27T11:20:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1352 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:20:34 dist-test-slave-lsvg krb5kdc[14079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:20:34 dist-test-slave-lsvg krb5kdc[14079](info): set up 2 sockets
Sep 27 11:20:34 dist-test-slave-lsvg krb5kdc[14079](info): commencing operation
krb5kdc: starting...
W20260927 11:20:35.629220 392 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:20:35 dist-test-slave-lsvg krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:20:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:20:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:20:35.747999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41253
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:35.828085 14092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:35.828308 14092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:35.828331 14092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:35.829434 14092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:20:35.829469 14092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:35.829484 14092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:35.829496 14092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:20:35.829509 14092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:20:35.830900 14092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:35.831095 14092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:35.831274 14092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:35.836005 14098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:35.836062 14100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:35.836076 14097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:35.836457 14092 server_base.cc:1034] running on GCE node
I20260927 11:20:35.836687 14092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:35.836956 14092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:35.838105 14092 hybrid_clock.cc:648] HybridClock initialized: now 1790508035838083 us; error 39 us; skew 500 ppm
Sep 27 11:20:35 dist-test-slave-lsvg krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:35.840502 14092 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:20:35.840941 14092 webserver.cc:460] Webserver started at http://127.0.98.62:46641/ using document root <none> and password file <none>
I20260927 11:20:35.841218 14092 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:35.841266 14092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:35.841411 14092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:20:35.842447 14092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "6e37444cc864412daa8ac39286643c24"
format_stamp: "Formatted at 2026-09-27 11:20:35 on dist-test-slave-lsvg"
server_key: "11c7f4d12d2324b4232ffd3dc80ff249"
server_key_iv: "e4a057045ae891958472d7fb0754ca19"
server_key_version: "encryptionkey@0"
I20260927 11:20:35.842810 14092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e37444cc864412daa8ac39286643c24"
format_stamp: "Formatted at 2026-09-27 11:20:35 on dist-test-slave-lsvg"
server_key: "11c7f4d12d2324b4232ffd3dc80ff249"
server_key_iv: "e4a057045ae891958472d7fb0754ca19"
server_key_version: "encryptionkey@0"
I20260927 11:20:35.844141 14092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:20:35.844842 14107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:35.845010 14092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:20:35.845129 14092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "6e37444cc864412daa8ac39286643c24"
format_stamp: "Formatted at 2026-09-27 11:20:35 on dist-test-slave-lsvg"
server_key: "11c7f4d12d2324b4232ffd3dc80ff249"
server_key_iv: "e4a057045ae891958472d7fb0754ca19"
server_key_version: "encryptionkey@0"
I20260927 11:20:35.845214 14092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:35.852353 14092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:35.854723 14092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:35.854902 14092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:35.858961 14092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41253
I20260927 11:20:35.858994 14159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41253 every 8 connection(s)
I20260927 11:20:35.859344 14092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:20:35.860278 14160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:35.862565 14160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Bootstrap starting.
I20260927 11:20:35.862870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 14092
I20260927 11:20:35.862995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:20:35.863212 392 external_mini_cluster.cc:1428] Setting key 3beddefb07090e9e0905d717e225d863
I20260927 11:20:35.863304 14160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:35.863615 14160 log.cc:824] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:35.864454 14160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: No bootstrap required, opened a new log
I20260927 11:20:35.866433 14160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260927 11:20:35.866600 14160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:35.866675 14160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e37444cc864412daa8ac39286643c24, State: Initialized, Role: FOLLOWER
I20260927 11:20:35.866849 14160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260927 11:20:35.866945 14160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:20:35.866989 14160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:20:35.867038 14160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:35.867599 14160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260927 11:20:35.867724 14160 leader_election.cc:302] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e37444cc864412daa8ac39286643c24; no voters:
I20260927 11:20:35.867923 14160 leader_election.cc:288] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:20:35.867959 14165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:20:35.868088 14165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 1 LEADER]: Becoming Leader. State: Replica: 6e37444cc864412daa8ac39286643c24, State: Running, Role: LEADER
I20260927 11:20:35.868227 14160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:20:35.868534 14165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260927 11:20:35.868903 14167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e37444cc864412daa8ac39286643c24. Latest consensus state: current_term: 1 leader_uuid: "6e37444cc864412daa8ac39286643c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } } }
I20260927 11:20:35.868991 14167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:35.869252 14166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e37444cc864412daa8ac39286643c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } } }
I20260927 11:20:35.869359 14166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:35.869637 14173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:20:35.870168 14173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 11:20:35 dist-test-slave-lsvg krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:35.871944 14173 catalog_manager.cc:1347] Generated new cluster ID: 8320e2d00d484d08b68195d0a2d039eb
I20260927 11:20:35.871990 14173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:20:35.873978 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:35.863775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49328 (local address 127.0.98.62:41253)
0927 11:20:35.863991 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:35.864000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:35.864043 (+ 43us) server_negotiation.cc:407] Connection header received
0927 11:20:35.864488 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:35.864518 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:35.864596 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:35.864752 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:35.865472 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:35.866346 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:35.867329 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:35.867781 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:35.867897 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:35.870688 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:35.870692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:35.870693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:35.870705 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:35.872401 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:35.872773 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:35.872776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:35.872777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:35.872867 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:35.873137 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:35.873138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:35.873139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:35.873362 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:35.873525 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:35.873630 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 11:20:35.873793 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":116,"threads_started":1}
I20260927 11:20:35.881302 14173 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:20:35.882020 14173 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:20:35.884943 14173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Generated new TSK 0
I20260927 11:20:35.885149 14173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:20:35.916452 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41253
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:35.998546 14186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:35.998751 14186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:35.998775 14186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:36.000106 14186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:36.000154 14186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:36.000180 14186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:20:36.001773 14186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:36.002030 14186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:36.002290 14186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:36.007458 14194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:36.007488 14192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:36.007527 14186 server_base.cc:1034] running on GCE node
W20260927 11:20:36.007488 14191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:36.007838 14186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:36.008093 14186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:36.009253 14186 hybrid_clock.cc:648] HybridClock initialized: now 1790508036009239 us; error 28 us; skew 500 ppm
Sep 27 11:20:36 dist-test-slave-lsvg krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:36.011742 14186 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:20:36.012158 14186 webserver.cc:460] Webserver started at http://127.0.98.1:42449/ using document root <none> and password file <none>
I20260927 11:20:36.012403 14186 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:36.012475 14186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:36.012617 14186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:20:36.013710 14186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "3e2375972ca6477f95198463edc4ff70"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "f52887a039153d5233a6326d22a96234"
server_key_iv: "01a3c026c34101b5c91be23d3f65e269"
server_key_version: "encryptionkey@0"
I20260927 11:20:36.014062 14186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3e2375972ca6477f95198463edc4ff70"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "f52887a039153d5233a6326d22a96234"
server_key_iv: "01a3c026c34101b5c91be23d3f65e269"
server_key_version: "encryptionkey@0"
I20260927 11:20:36.015340 14186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:20:36.016016 14201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.016197 14186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:20:36.016287 14186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "3e2375972ca6477f95198463edc4ff70"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "f52887a039153d5233a6326d22a96234"
server_key_iv: "01a3c026c34101b5c91be23d3f65e269"
server_key_version: "encryptionkey@0"
I20260927 11:20:36.016366 14186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:36.026368 14186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:36.028597 14186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:36.028759 14186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:36.028982 14186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:36.029372 14186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:36.029428 14186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.029466 14186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:36.029493 14186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.035634 14186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42543
I20260927 11:20:36.035723 14314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42543 every 8 connection(s)
I20260927 11:20:36.036047 14186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:20:36.041270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 14186
I20260927 11:20:36.041388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:20:36.041594 392 external_mini_cluster.cc:1428] Setting key df02ad8a133f1778198c18470883481e
Sep 27 11:20:36 dist-test-slave-lsvg krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:36.045632 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.036831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:47613)
0927 11:20:36.037019 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:36.037094 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:36.037576 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:36.037800 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:36.037804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:36.038367 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:36.038563 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.038568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.039214 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.039214 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.040013 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.040015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.040356 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.040357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.040452 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.040989 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:36.040998 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:36.042605 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:36.044208 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:36.044215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:36.044217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:36.044519 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:36.044762 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:36.044762 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:36.044763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:36.044858 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:36.045153 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:36.045157 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:36.045271 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 11:20:36.045351 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 11:20:36.045469 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":109,"threads_started":1}
I20260927 11:20:36.045652 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.036866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47613 (local address 127.0.98.62:41253)
0927 11:20:36.036953 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:36.036956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:36.037187 (+ 231us) server_negotiation.cc:407] Connection header received
0927 11:20:36.037661 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:36.037662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:36.037671 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:36.037737 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:36.038641 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.039134 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.040086 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.040286 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.040336 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.042695 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:36.042696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:36.042696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:36.042702 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:36.044121 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:36.044627 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:36.044628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:36.044629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:36.044691 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:36.044919 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:36.044920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:36.044920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:36.045092 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:36.045154 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:36.045399 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 11:20:36.045508 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 11:20:36.046408 14315 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41253
I20260927 11:20:36.046490 14315 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:36.046674 14315 heartbeater.cc:511] Master 127.0.98.62:41253 requested a full tablet report, sending...
I20260927 11:20:36.047237 14124 ts_manager.cc:194] Registered new tserver with Master: 3e2375972ca6477f95198463edc4ff70 (127.0.98.1:42543)
I20260927 11:20:36.047940 14124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47613
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:20:36.067651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41253
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:36.150625 14320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:36.150866 14320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:36.150905 14320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:36.152038 14320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:36.152102 14320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:36.152146 14320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:20:36.153643 14320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:36.153898 14320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:36.154125 14320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:36.159221 14328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:36.159230 14326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:36.159475 14320 server_base.cc:1034] running on GCE node
W20260927 11:20:36.159615 14325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:36.159842 14320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:36.160089 14320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:36.161232 14320 hybrid_clock.cc:648] HybridClock initialized: now 1790508036161226 us; error 37 us; skew 500 ppm
Sep 27 11:20:36 dist-test-slave-lsvg krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:36.168525 14320 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:20:36.169010 14320 webserver.cc:460] Webserver started at http://127.0.98.2:37049/ using document root <none> and password file <none>
I20260927 11:20:36.169289 14320 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:36.169359 14320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:36.169497 14320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:20:36.170612 14320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "bd6ce8a0153f473db53476f14a5b4544"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "21fb9d47694ac632de50781fb0ec5ed8"
server_key_iv: "05c03c18cb7d820c20bcb564f87ba2f7"
server_key_version: "encryptionkey@0"
I20260927 11:20:36.170976 14320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bd6ce8a0153f473db53476f14a5b4544"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "21fb9d47694ac632de50781fb0ec5ed8"
server_key_iv: "05c03c18cb7d820c20bcb564f87ba2f7"
server_key_version: "encryptionkey@0"
I20260927 11:20:36.172416 14320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:20:36.173230 14335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.173427 14320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:20:36.173527 14320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "bd6ce8a0153f473db53476f14a5b4544"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "21fb9d47694ac632de50781fb0ec5ed8"
server_key_iv: "05c03c18cb7d820c20bcb564f87ba2f7"
server_key_version: "encryptionkey@0"
I20260927 11:20:36.173612 14320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:36.186404 14320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:36.188730 14320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:36.188889 14320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:36.189188 14320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:36.189517 14320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:36.189570 14320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.189610 14320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:36.189641 14320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.195389 14320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37513
I20260927 11:20:36.195441 14448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37513 every 8 connection(s)
I20260927 11:20:36.195742 14320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:20:36 dist-test-slave-lsvg krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:36.202827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 14320
I20260927 11:20:36.202955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:20:36.203153 392 external_mini_cluster.cc:1428] Setting key 0bd1b76d4360ec18f47a52359ac674f2
I20260927 11:20:36.205298 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.196424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48311 (local address 127.0.98.62:41253)
0927 11:20:36.196466 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:36.196468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:36.196684 (+ 216us) server_negotiation.cc:407] Connection header received
0927 11:20:36.197192 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:36.197193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:36.197240 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:36.197294 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:36.198161 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.198663 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.199622 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.199812 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.199868 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.202133 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:36.202135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:36.202135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:36.202143 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:36.203769 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:36.204272 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:36.204274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:36.204274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:36.204330 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:36.204628 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:36.204629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:36.204629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:36.204738 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:36.204795 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:36.205155 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 11:20:36.205234 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:20:36.205588 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.196415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:48311)
0927 11:20:36.196559 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:36.196593 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:36.197099 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:36.197344 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:36.197347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:36.197889 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:36.198076 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.198081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.198746 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.198747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.199534 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.199536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.199886 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.199887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.199982 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.200493 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:36.200502 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:36.201964 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:36.203852 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:36.203858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:36.203860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:36.204192 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:36.204396 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:36.204397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:36.204398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:36.204551 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:36.204804 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:36.204810 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:36.204983 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 11:20:36.205126 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 11:20:36.205305 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260927 11:20:36.206252 14449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41253
I20260927 11:20:36.206347 14449 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:36.206542 14449 heartbeater.cc:511] Master 127.0.98.62:41253 requested a full tablet report, sending...
I20260927 11:20:36.206911 14124 ts_manager.cc:194] Registered new tserver with Master: bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513)
I20260927 11:20:36.207497 14124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48311
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:20:36.228766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41253
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:36.325034 14454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:36.325366 14454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:36.325405 14454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:36.327222 14454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:36.327282 14454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:36.327323 14454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:20:36.328837 14454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:36.329129 14454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:36.329384 14454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:36.334406 14459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:36.334401 14462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:36.334401 14460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:36.334774 14454 server_base.cc:1034] running on GCE node
I20260927 11:20:36.334977 14454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:36.335240 14454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:36.336385 14454 hybrid_clock.cc:648] HybridClock initialized: now 1790508036336376 us; error 31 us; skew 500 ppm
Sep 27 11:20:36 dist-test-slave-lsvg krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:36.338806 14454 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:20:36.339280 14454 webserver.cc:460] Webserver started at http://127.0.98.3:44179/ using document root <none> and password file <none>
I20260927 11:20:36.339517 14454 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:36.339588 14454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:36.339746 14454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:20:36.340818 14454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "1182b7f6a0c04d77a45cb23f5dd6117e"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "70eeaba52ce6b49fcfd5beee83889abd"
server_key_iv: "037db0f5c840c864d48ed8f044871ac1"
server_key_version: "encryptionkey@0"
I20260927 11:20:36.341266 14454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1182b7f6a0c04d77a45cb23f5dd6117e"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "70eeaba52ce6b49fcfd5beee83889abd"
server_key_iv: "037db0f5c840c864d48ed8f044871ac1"
server_key_version: "encryptionkey@0"
I20260927 11:20:36.342617 14454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:20:36.343338 14469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.343514 14454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:20:36.343614 14454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "1182b7f6a0c04d77a45cb23f5dd6117e"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "70eeaba52ce6b49fcfd5beee83889abd"
server_key_iv: "037db0f5c840c864d48ed8f044871ac1"
server_key_version: "encryptionkey@0"
I20260927 11:20:36.343704 14454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:36.368705 14454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:36.371779 14454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:36.371912 14454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:36.372119 14454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:36.372406 14454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:36.372442 14454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.372467 14454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:36.372481 14454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.379156 14454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37195
I20260927 11:20:36.379233 14582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37195 every 8 connection(s)
I20260927 11:20:36.379604 14454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:20:36.384498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 14454
I20260927 11:20:36.384609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:20:36.384817 392 external_mini_cluster.cc:1428] Setting key 5ac4818f06cc9eb5e5ff94c4a9a2b097
Sep 27 11:20:36 dist-test-slave-lsvg krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:36.389142 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.380379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54859 (local address 127.0.98.62:41253)
0927 11:20:36.380458 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:36.380460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:36.380654 (+ 194us) server_negotiation.cc:407] Connection header received
0927 11:20:36.381238 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:36.381239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:36.381251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:36.381307 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:36.382264 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.382773 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.383768 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.383988 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.384065 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.386702 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:36.386703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:36.386714 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:36.386721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:36.387901 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:36.388304 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:36.388305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:36.388305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:36.388354 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:36.388575 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:36.388576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:36.388576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:36.388672 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:36.388731 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:36.388950 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 11:20:36.389007 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 11:20:36.389389 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.380359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:54859)
0927 11:20:36.380530 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:36.380571 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:36.381123 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:36.381376 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:36.381380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:36.381953 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:36.382152 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.382157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.382867 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.382868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.383675 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.383677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.384069 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.384070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.384209 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.384924 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:36.384935 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:36.386606 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:36.387984 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:36.387990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:36.387992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:36.388234 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:36.388416 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:36.388417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:36.388417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:36.388519 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:36.388729 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:36.388733 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:36.388846 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 11:20:36.388926 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 11:20:36.389118 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260927 11:20:36.389899 14583 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41253
I20260927 11:20:36.389994 14583 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:36.390128 14583 heartbeater.cc:511] Master 127.0.98.62:41253 requested a full tablet report, sending...
I20260927 11:20:36.390484 14124 ts_manager.cc:194] Registered new tserver with Master: 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195)
I20260927 11:20:36.390874 14124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54859
I20260927 11:20:36.396672 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:20:36.404121 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.398124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.0.98.62:41253)
0927 11:20:36.398172 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:36.398174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:36.398183 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:20:36.398237 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:36.398238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:36.398248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:36.398303 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:36.398872 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.399356 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.400081 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.400297 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.400381 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.401684 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:36.401686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:36.401686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:36.401692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:36.402904 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:36.403259 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:36.403260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:36.403260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:36.403321 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:36.403519 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:36.403520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:36.403520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:36.403672 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:36.403788 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:36.403939 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 11:20:36.404022 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 11:20:36.405673 14124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:20:36.407202 14124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:20:36.413736 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.409550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37195 (local address 127.0.0.1:46878)
0927 11:20:36.409823 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:36.409866 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:36.409944 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:36.410239 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:36.410242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:36.410245 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:36.410312 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.410313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.411339 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.411340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.412533 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.412536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.413441 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.413442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.413520 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.413536 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 11:20:36.413612 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 11:20:36.413662 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":52,"threads_started":1}
I20260927 11:20:36.413789 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.409635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.0.98.3:37195)
0927 11:20:36.409801 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:36.409807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:36.409941 (+ 134us) server_negotiation.cc:407] Connection header received
0927 11:20:36.410087 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:36.410089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:36.410107 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:36.410194 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:36.410461 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.411238 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.412605 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.413356 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.413432 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.413489 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:36.413640 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 11:20:36.413688 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260927 11:20:36.413997 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.409595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42543 (local address 127.0.0.1:59152)
0927 11:20:36.409955 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:36.409964 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:36.410003 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:36.410303 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:36.410304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:36.410305 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:36.410355 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.410356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.411760 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.411761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.412802 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.412803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.413752 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.413753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.413829 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.413834 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:20:36.413867 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 11:20:36.413914 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":87,"threads_started":1}
I20260927 11:20:36.414465 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.409704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59152 (local address 127.0.98.1:42543)
0927 11:20:36.410118 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:36.410122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:36.410132 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:20:36.410199 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:36.410202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:36.410227 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:36.410308 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:36.410406 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.411639 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.412872 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.413656 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.414253 (+ 597us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.414327 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:36.414352 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 11:20:36.414390 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":72,"threads_started":1}
I20260927 11:20:36.414633 14517 tablet_service.cc:1467] Processing CreateTablet for tablet 55d3766bc53f4e7c833de9d675231332 (DEFAULT_TABLE table=test-table [id=da6046fdbdf647d3bf5897441c5c0362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:20:36.414909 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.409755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37513 (local address 127.0.0.1:47710)
0927 11:20:36.410071 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:36.410079 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:36.410112 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:36.410385 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:36.410386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:36.410388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:36.410454 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.410456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.412869 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.412869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.413770 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.413773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.414676 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.414677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.414754 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.414771 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 11:20:36.414806 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 11:20:36.414846 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":64,"threads_started":1}
I20260927 11:20:36.415030 14517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d3766bc53f4e7c833de9d675231332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:36.415223 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.410001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.0.98.2:37513)
0927 11:20:36.410223 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:36.410227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:36.410236 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:20:36.410274 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:36.410276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:36.410291 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:36.410533 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:36.410605 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.411649 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.413866 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.414573 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.415009 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.415073 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:36.415100 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 11:20:36.415142 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260927 11:20:36.415169 14249 tablet_service.cc:1467] Processing CreateTablet for tablet 55d3766bc53f4e7c833de9d675231332 (DEFAULT_TABLE table=test-table [id=da6046fdbdf647d3bf5897441c5c0362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:20:36.415468 14249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d3766bc53f4e7c833de9d675231332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:36.417798 14600 tablet_bootstrap.cc:492] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e: Bootstrap starting.
I20260927 11:20:36.417778 14383 tablet_service.cc:1467] Processing CreateTablet for tablet 55d3766bc53f4e7c833de9d675231332 (DEFAULT_TABLE table=test-table [id=da6046fdbdf647d3bf5897441c5c0362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:20:36.418123 14383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d3766bc53f4e7c833de9d675231332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:36.418133 14601 tablet_bootstrap.cc:492] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Bootstrap starting.
I20260927 11:20:36.418568 14600 tablet_bootstrap.cc:654] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:36.418781 14601 tablet_bootstrap.cc:654] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:36.419129 14600 log.cc:824] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:36.419329 14601 log.cc:824] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:36.419988 14600 tablet_bootstrap.cc:492] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e: No bootstrap required, opened a new log
I20260927 11:20:36.420081 14600 ts_tablet_manager.cc:1397] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:20:36.420105 14601 tablet_bootstrap.cc:492] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: No bootstrap required, opened a new log
I20260927 11:20:36.420163 14601 ts_tablet_manager.cc:1397] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:20:36.421676 14604 tablet_bootstrap.cc:492] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544: Bootstrap starting.
I20260927 11:20:36.421631 14600 raft_consensus.cc:348] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:36.421818 14600 raft_consensus.cc:374] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:36.421880 14600 raft_consensus.cc:729] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1182b7f6a0c04d77a45cb23f5dd6117e, State: Initialized, Role: FOLLOWER
I20260927 11:20:36.422015 14600 consensus_queue.cc:260] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:36.422319 14600 ts_tablet_manager.cc:1428] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 11:20:36.422412 14604 tablet_bootstrap.cc:654] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:36.422286 14601 raft_consensus.cc:348] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:36.422456 14601 raft_consensus.cc:374] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:36.422462 14583 heartbeater.cc:503] Master 127.0.98.62:41253 was elected leader, sending a full tablet report...
I20260927 11:20:36.422503 14601 raft_consensus.cc:729] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2375972ca6477f95198463edc4ff70, State: Initialized, Role: FOLLOWER
I20260927 11:20:36.422602 14601 consensus_queue.cc:260] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:36.422868 14601 ts_tablet_manager.cc:1428] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:20:36.422994 14604 log.cc:824] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:36.423084 14315 heartbeater.cc:503] Master 127.0.98.62:41253 was elected leader, sending a full tablet report...
I20260927 11:20:36.423954 14604 tablet_bootstrap.cc:492] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544: No bootstrap required, opened a new log
I20260927 11:20:36.424094 14604 ts_tablet_manager.cc:1397] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:20:36.425539 14604 raft_consensus.cc:348] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:36.425690 14604 raft_consensus.cc:374] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:36.425734 14604 raft_consensus.cc:729] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6ce8a0153f473db53476f14a5b4544, State: Initialized, Role: FOLLOWER
I20260927 11:20:36.425850 14604 consensus_queue.cc:260] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:36.426098 14604 ts_tablet_manager.cc:1428] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:20:36.426177 14449 heartbeater.cc:503] Master 127.0.98.62:41253 was elected leader, sending a full tablet report...
W20260927 11:20:36.446271 14450 tablet.cc:2369] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:20:36.510679 14605 raft_consensus.cc:482] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:20:36.510814 14605 raft_consensus.cc:504] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:36.511365 14605 leader_election.cc:288] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513), 3e2375972ca6477f95198463edc4ff70 (127.0.98.1:42543)
I20260927 11:20:36.514375 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.511722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45577 (local address 127.0.98.1:42543)
0927 11:20:36.511769 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:36.511771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:36.511779 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:20:36.511809 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:36.511810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:36.511821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:36.511880 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:36.512083 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.512608 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.513634 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.514017 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.514072 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.514079 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:36.514261 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 11:20:36.514303 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 11:20:36.514393 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.511710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45723 (local address 127.0.98.2:37513)
0927 11:20:36.511762 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:36.511764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:36.511909 (+ 145us) server_negotiation.cc:407] Connection header received
0927 11:20:36.511936 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:36.511937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:36.511950 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:36.512007 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:36.512198 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.512662 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.513674 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.514031 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.514120 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.514129 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:36.514292 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 11:20:36.514331 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 11:20:36.514382 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.511553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42543 (local address 127.0.98.3:45577)
0927 11:20:36.511597 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:36.511610 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:36.511675 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:36.511924 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:36.511925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:36.511927 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:36.512002 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.512004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.512692 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.512693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.513544 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.513546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.514099 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.514100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.514181 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.514189 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 11:20:36.514233 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 11:20:36.514292 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":2}
I20260927 11:20:36.514398 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.511553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37513 (local address 127.0.98.3:45723)
0927 11:20:36.511787 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:36.511798 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:36.511848 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:36.512025 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:36.512026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:36.512027 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:36.512123 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.512124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.512749 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.512750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.513558 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.513560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.514136 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.514137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.514198 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.514202 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:20:36.514273 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:20:36.514306 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":16,"thread_start_us":132,"threads_started":1}
I20260927 11:20:36.514887 14269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d3766bc53f4e7c833de9d675231332" candidate_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2375972ca6477f95198463edc4ff70" is_pre_election: true
I20260927 11:20:36.514871 14403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d3766bc53f4e7c833de9d675231332" candidate_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6ce8a0153f473db53476f14a5b4544" is_pre_election: true
I20260927 11:20:36.515038 14403 raft_consensus.cc:2454] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1182b7f6a0c04d77a45cb23f5dd6117e in term 0.
I20260927 11:20:36.515038 14269 raft_consensus.cc:2454] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1182b7f6a0c04d77a45cb23f5dd6117e in term 0.
I20260927 11:20:36.515271 14470 leader_election.cc:302] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1182b7f6a0c04d77a45cb23f5dd6117e, bd6ce8a0153f473db53476f14a5b4544; no voters:
I20260927 11:20:36.515424 14605 raft_consensus.cc:2781] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:20:36.515473 14605 raft_consensus.cc:482] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:20:36.515496 14605 raft_consensus.cc:3037] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:36.516258 14605 raft_consensus.cc:504] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:36.516403 14605 leader_election.cc:288] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [CANDIDATE]: Term 1 election: Requested vote from peers bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513), 3e2375972ca6477f95198463edc4ff70 (127.0.98.1:42543)
I20260927 11:20:36.516566 14403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d3766bc53f4e7c833de9d675231332" candidate_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6ce8a0153f473db53476f14a5b4544"
I20260927 11:20:36.516609 14269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d3766bc53f4e7c833de9d675231332" candidate_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2375972ca6477f95198463edc4ff70"
I20260927 11:20:36.516664 14403 raft_consensus.cc:3037] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:36.516676 14269 raft_consensus.cc:3037] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:36.517608 14269 raft_consensus.cc:2454] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1182b7f6a0c04d77a45cb23f5dd6117e in term 1.
I20260927 11:20:36.517608 14403 raft_consensus.cc:2454] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1182b7f6a0c04d77a45cb23f5dd6117e in term 1.
I20260927 11:20:36.517825 14473 leader_election.cc:302] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1182b7f6a0c04d77a45cb23f5dd6117e, 3e2375972ca6477f95198463edc4ff70; no voters:
I20260927 11:20:36.517967 14605 raft_consensus.cc:2781] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:20:36.518127 14605 raft_consensus.cc:686] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 1 LEADER]: Becoming Leader. State: Replica: 1182b7f6a0c04d77a45cb23f5dd6117e, State: Running, Role: LEADER
I20260927 11:20:36.518232 14605 consensus_queue.cc:237] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:36.519101 14124 catalog_manager.cc:5495] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } health_report { overall_health: UNKNOWN } } }
I20260927 11:20:36.531724 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.529152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.0.98.3:37195)
0927 11:20:36.529230 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:36.529232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:36.529249 (+ 17us) server_negotiation.cc:407] Connection header received
0927 11:20:36.529316 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:36.529317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:36.529339 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:36.529408 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:20:36.529613 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.530111 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.530896 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.531089 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.531160 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.531322 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:20:36.531417 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:20:36.531485 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:36.531556 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 11:20:36.531576 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 11:20:36.533543 14269 raft_consensus.cc:1261] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Refusing update from remote peer 1182b7f6a0c04d77a45cb23f5dd6117e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:20:36.533551 14403 raft_consensus.cc:1261] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544 [term 1 FOLLOWER]: Refusing update from remote peer 1182b7f6a0c04d77a45cb23f5dd6117e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:20:36.533923 14610 consensus_queue.cc:1035] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:20:36.534066 14605 consensus_queue.cc:1035] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:20:36.536125 14613 mvcc.cc:205] Tried to move back new op lower bound from 7333920917638557696 to 7333920917579317248. Current Snapshot: MvccSnapshot[applied={T|T < 7333920917638557696}]
I20260927 11:20:36.536372 14614 mvcc.cc:205] Tried to move back new op lower bound from 7333920917638557696 to 7333920917579317248. Current Snapshot: MvccSnapshot[applied={T|T < 7333920917638557696}]
I20260927 11:20:36.538028 14616 mvcc.cc:205] Tried to move back new op lower bound from 7333920917638557696 to 7333920917579317248. Current Snapshot: MvccSnapshot[applied={T|T < 7333920917638557696 or (T in {7333920917638557696})}]
I20260927 11:20:36.538962 14124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:20:36.539069 14124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:20:36.539888 14124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Sending DeleteTablet for 3 replicas of tablet 55d3766bc53f4e7c833de9d675231332
I20260927 11:20:36.540192 14249 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d3766bc53f4e7c833de9d675231332 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59152
I20260927 11:20:36.540283 14517 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d3766bc53f4e7c833de9d675231332 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46878
I20260927 11:20:36.540287 14383 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d3766bc53f4e7c833de9d675231332 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47710
I20260927 11:20:36.540498 14623 tablet_replica.cc:330] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e: stopping tablet replica
I20260927 11:20:36.540598 14623 raft_consensus.cc:2229] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:20:36.540597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 14186
I20260927 11:20:36.541594 14623 raft_consensus.cc:2258] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:20:36.542076 14623 ts_tablet_manager.cc:1905] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:20:36.543767 14624 tablet_replica.cc:330] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544: stopping tablet replica
I20260927 11:20:36.543883 14623 ts_tablet_manager.cc:1918] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:20:36.543922 14624 raft_consensus.cc:2229] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:20:36.544036 14623 log.cc:1196] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/55d3766bc53f4e7c833de9d675231332
I20260927 11:20:36.544049 14624 raft_consensus.cc:2258] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:20:36.544402 14623 ts_tablet_manager.cc:1939] T 55d3766bc53f4e7c833de9d675231332 P 1182b7f6a0c04d77a45cb23f5dd6117e: Deleting consensus metadata
I20260927 11:20:36.544482 14624 ts_tablet_manager.cc:1905] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:20:36.544838 14108 catalog_manager.cc:4841] TS 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195): tablet 55d3766bc53f4e7c833de9d675231332 (table test-table [id=da6046fdbdf647d3bf5897441c5c0362]) successfully deleted
I20260927 11:20:36.546206 14624 ts_tablet_manager.cc:1918] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:20:36.546283 14624 log.cc:1196] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/55d3766bc53f4e7c833de9d675231332
I20260927 11:20:36.546655 14624 ts_tablet_manager.cc:1939] T 55d3766bc53f4e7c833de9d675231332 P bd6ce8a0153f473db53476f14a5b4544: Deleting consensus metadata
W20260927 11:20:36.546940 14111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42543 (error 108)
I20260927 11:20:36.547057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 14320
I20260927 11:20:36.547662 14108 catalog_manager.cc:4841] TS bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513): tablet 55d3766bc53f4e7c833de9d675231332 (table test-table [id=da6046fdbdf647d3bf5897441c5c0362]) successfully deleted
I20260927 11:20:36.548107 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.547967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42543 (local address 127.0.0.1:59158)
0927 11:20:36.548033 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:36.548075 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 11:20:36.548259 14111 catalog_manager.cc:4584] TS 3e2375972ca6477f95198463edc4ff70 (127.0.98.1:42543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55d3766bc53f4e7c833de9d675231332: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42543: connect: Connection refused (error 111)
I20260927 11:20:36.552789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 14454
I20260927 11:20:36.557842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 14092
I20260927 11:20:36.563035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41253
--webserver_interface=127.0.98.62
--webserver_port=46641
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:36.643242 14626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:36.643460 14626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:36.643482 14626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:36.644685 14626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:20:36.644730 14626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:36.644744 14626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:36.644757 14626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:20:36.644770 14626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:20:36.646364 14626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:36.646627 14626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:36.646884 14626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:36.655398 14635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:36.655481 14633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:36.655414 14632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:36.656049 14626 server_base.cc:1034] running on GCE node
I20260927 11:20:36.656304 14626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:36.656608 14626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:36.657773 14626 hybrid_clock.cc:648] HybridClock initialized: now 1790508036657757 us; error 49 us; skew 500 ppm
Sep 27 11:20:36 dist-test-slave-lsvg krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:36.663887 14626 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:20:36.664388 14626 webserver.cc:460] Webserver started at http://127.0.98.62:46641/ using document root <none> and password file <none>
I20260927 11:20:36.664592 14626 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:36.664647 14626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:36.666050 14626 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:36.666682 14642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.666872 14626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:20:36.666957 14626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "6e37444cc864412daa8ac39286643c24"
format_stamp: "Formatted at 2026-09-27 11:20:35 on dist-test-slave-lsvg"
server_key: "11c7f4d12d2324b4232ffd3dc80ff249"
server_key_iv: "e4a057045ae891958472d7fb0754ca19"
server_key_version: "encryptionkey@0"
I20260927 11:20:36.667385 14626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:36.691377 14626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:36.693567 14626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:36.693681 14626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:36.697513 14626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41253
I20260927 11:20:36.697542 14694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41253 every 8 connection(s)
I20260927 11:20:36.697959 14626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:20:36.699585 14695 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:20:36.700284 14695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Bootstrap starting.
I20260927 11:20:36.707984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 14626
I20260927 11:20:36.708298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42543
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41253
--webserver_port=42449
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:20:36.708668 14695 log.cc:824] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:36.710712 14695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:20:36.711006 14695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Bootstrap complete.
I20260927 11:20:36.713356 14695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260927 11:20:36.713590 14695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e37444cc864412daa8ac39286643c24, State: Initialized, Role: FOLLOWER
I20260927 11:20:36.713768 14695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260927 11:20:36.713831 14695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:20:36.713883 14695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:20:36.713934 14695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:20:36.714979 14695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260927 11:20:36.715070 14695 leader_election.cc:302] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e37444cc864412daa8ac39286643c24; no voters:
I20260927 11:20:36.715297 14695 leader_election.cc:288] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 11:20:36.715332 14699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 11:20:36.715456 14699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 2 LEADER]: Becoming Leader. State: Replica: 6e37444cc864412daa8ac39286643c24, State: Running, Role: LEADER
I20260927 11:20:36.715564 14695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:20:36.715610 14699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260927 11:20:36.715962 14699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e37444cc864412daa8ac39286643c24. Latest consensus state: current_term: 2 leader_uuid: "6e37444cc864412daa8ac39286643c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } } }
I20260927 11:20:36.715965 14701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6e37444cc864412daa8ac39286643c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } } }
I20260927 11:20:36.716143 14699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:36.716245 14701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:36.716816 14708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:20:36.717697 14708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:20:36.717803 14708 catalog_manager.cc:1259] Loaded cluster ID: 8320e2d00d484d08b68195d0a2d039eb
I20260927 11:20:36.717835 14708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:20:36.718052 14708 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 11:20:36.722694 14708 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 11:20:36.725828 14708 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:20:36.726040 14708 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Loaded TSK: 0
I20260927 11:20:36.726279 14708 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 11:20:36.790894 14698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:36.791145 14698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:36.791190 14698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:36.792388 14698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:36.792459 14698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:36.792505 14698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:20:36.794059 14698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42449
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:36.794355 14698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:36.794595 14698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:36.799888 14723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:36.799918 14722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:36.799934 14725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:36.800314 14698 server_base.cc:1034] running on GCE node
I20260927 11:20:36.800542 14698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:36.800863 14698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:36.802047 14698 hybrid_clock.cc:648] HybridClock initialized: now 1790508036802027 us; error 36 us; skew 500 ppm
Sep 27 11:20:36 dist-test-slave-lsvg krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:36.805088 14698 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:20:36.805609 14698 webserver.cc:460] Webserver started at http://127.0.98.1:42449/ using document root <none> and password file <none>
I20260927 11:20:36.805894 14698 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:36.805971 14698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:36.807751 14698 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:20:36.808512 14732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.808708 14698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:36.808804 14698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "3e2375972ca6477f95198463edc4ff70"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "f52887a039153d5233a6326d22a96234"
server_key_iv: "01a3c026c34101b5c91be23d3f65e269"
server_key_version: "encryptionkey@0"
I20260927 11:20:36.809178 14698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:36.818779 14698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:36.821090 14698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:36.821264 14698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:36.821492 14698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:36.821947 14739 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 11:20:36.823837 14698 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 11:20:36.823899 14698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 11:20:36.823956 14698 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 11:20:36.824575 14698 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 11:20:36.824630 14698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 11:20:36.824707 14739 tablet_bootstrap.cc:492] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Bootstrap starting.
I20260927 11:20:36.832015 14698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42543
I20260927 11:20:36.832425 14698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:20:36.833506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 14698
I20260927 11:20:36.833781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37513
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41253
--webserver_port=37049
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:20:36.835362 14739 log.cc:824] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:36.837699 14739 tablet_bootstrap.cc:492] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:20:36.838021 14739 tablet_bootstrap.cc:492] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Bootstrap complete.
I20260927 11:20:36.838955 14739 ts_tablet_manager.cc:1397] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260927 11:20:36.839954 14739 raft_consensus.cc:348] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:36.840199 14739 raft_consensus.cc:729] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2375972ca6477f95198463edc4ff70, State: Initialized, Role: FOLLOWER
I20260927 11:20:36.840366 14739 consensus_queue.cc:260] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:36.840430 14846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42543 every 8 connection(s)
I20260927 11:20:36.840796 14739 ts_tablet_manager.cc:1428] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Sep 27 11:20:36 dist-test-slave-lsvg krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:36.849774 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.835133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32993 (local address 127.0.98.62:41253)
0927 11:20:36.840451 (+ 5318us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:36.840460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:36.840507 (+ 47us) server_negotiation.cc:407] Connection header received
0927 11:20:36.840649 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:36.840655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:36.840712 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:36.840858 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:36.841597 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.842489 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.843515 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.843933 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.844066 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.846353 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:36.846358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:36.846358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:36.846368 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:36.847996 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:36.848455 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:36.848458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:36.848458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:36.848552 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:36.848814 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:36.848815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:36.848816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:36.849032 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:36.849156 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:36.849459 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 11:20:36.849581 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5259,"thread_start_us":88,"threads_started":1}
I20260927 11:20:36.849774 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.833308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:32993)
0927 11:20:36.835791 (+ 2483us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:36.835837 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:36.836437 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:36.840911 (+ 4474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:36.840915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:36.841278 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:36.841467 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.841472 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.842609 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.842611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.843418 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.843420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.844041 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.844042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.844173 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.844756 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:36.844764 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:36.846197 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:36.848100 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:36.848125 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:36.848127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:36.848369 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:36.848624 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:36.848625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:36.848626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:36.848743 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:36.849147 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:36.849153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:36.849291 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 11:20:36.849394 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 11:20:36.849566 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2441,"thread_start_us":110,"threads_started":1}
I20260927 11:20:36.850840 14847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41253
I20260927 11:20:36.850960 14847 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:36.851148 14847 heartbeater.cc:511] Master 127.0.98.62:41253 requested a full tablet report, sending...
I20260927 11:20:36.851920 14659 ts_manager.cc:194] Registered new tserver with Master: 3e2375972ca6477f95198463edc4ff70 (127.0.98.1:42543)
I20260927 11:20:36.853159 14659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32993
I20260927 11:20:36.854007 14847 heartbeater.cc:503] Master 127.0.98.62:41253 was elected leader, sending a full tablet report...
I20260927 11:20:36.860291 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.852785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.0.98.1:42543)
0927 11:20:36.852975 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:36.852980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:36.857305 (+ 4325us) server_negotiation.cc:407] Connection header received
0927 11:20:36.857407 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:36.857410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:36.857425 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:36.857499 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:36.857647 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.858389 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.859383 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.859865 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.859949 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.860005 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:36.860176 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 11:20:36.860215 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":128,"threads_started":1}
I20260927 11:20:36.860291 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.852516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42543 (local address 127.0.0.1:59174)
0927 11:20:36.857180 (+ 4664us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:36.857223 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:36.857327 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:36.857509 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:36.857512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:36.857513 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:36.857579 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.857581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.858478 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.858480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.859292 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.859295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.859962 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.859963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.860048 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.860060 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 11:20:36.860158 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 11:20:36.860214 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4626,"thread_start_us":111,"threads_started":1}
I20260927 11:20:36.860926 14781 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d3766bc53f4e7c833de9d675231332 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59174
I20260927 11:20:36.861528 14856 tablet_replica.cc:330] stopping tablet replica
I20260927 11:20:36.861653 14856 raft_consensus.cc:2229] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:20:36.861727 14856 raft_consensus.cc:2258] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:20:36.862033 14856 ts_tablet_manager.cc:1905] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:20:36.863965 14856 ts_tablet_manager.cc:1918] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:20:36.864050 14856 log.cc:1196] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal/wals/55d3766bc53f4e7c833de9d675231332
I20260927 11:20:36.864285 14856 ts_tablet_manager.cc:1939] T 55d3766bc53f4e7c833de9d675231332 P 3e2375972ca6477f95198463edc4ff70: Deleting consensus metadata
I20260927 11:20:36.864769 14646 catalog_manager.cc:4841] TS 3e2375972ca6477f95198463edc4ff70 (127.0.98.1:42543): tablet 55d3766bc53f4e7c833de9d675231332 (table test-table [id=da6046fdbdf647d3bf5897441c5c0362]) successfully deleted
W20260927 11:20:36.924140 14850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:36.924353 14850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:36.924379 14850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:36.925604 14850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:36.925657 14850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:36.925685 14850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:20:36.927161 14850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37049
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:36.927387 14850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:36.927611 14850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:20:36.932780 14850 server_base.cc:1034] running on GCE node
W20260927 11:20:36.932723 14862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:36.932747 14864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:36.932736 14861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:36.933214 14850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:36.933475 14850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:36.934619 14850 hybrid_clock.cc:648] HybridClock initialized: now 1790508036934610 us; error 32 us; skew 500 ppm
Sep 27 11:20:36 dist-test-slave-lsvg krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:36.937023 14850 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:20:36.937484 14850 webserver.cc:460] Webserver started at http://127.0.98.2:37049/ using document root <none> and password file <none>
I20260927 11:20:36.937662 14850 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:36.937716 14850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:36.938982 14850 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:36.939597 14871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.939765 14850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:20:36.939868 14850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "bd6ce8a0153f473db53476f14a5b4544"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "21fb9d47694ac632de50781fb0ec5ed8"
server_key_iv: "05c03c18cb7d820c20bcb564f87ba2f7"
server_key_version: "encryptionkey@0"
I20260927 11:20:36.940217 14850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:36.969406 14850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:36.971524 14850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:36.971685 14850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:36.971939 14850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:36.972252 14850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:36.972306 14850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.972349 14850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:36.972383 14850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:36.978344 14850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37513
I20260927 11:20:36.978415 14984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37513 every 8 connection(s)
I20260927 11:20:36.978775 14850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:20:36.979908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 14850
I20260927 11:20:36.980196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37195
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41253
--webserver_port=44179
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 11:20:36 dist-test-slave-lsvg krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:36.989686 14987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.979522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:46583)
0927 11:20:36.979679 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:36.979723 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:36.980458 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:36.980738 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:36.980742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:36.981612 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:36.981875 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.981881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.982905 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.982906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.983717 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:36.983719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.984311 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.984312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:36.984412 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.984977 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:36.984986 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:36.986407 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:36.988318 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:36.988324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:36.988326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:36.988545 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:36.988766 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:36.988766 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:36.988766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:36.988864 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:36.989176 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:36.989181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:36.989311 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 11:20:36.989396 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 11:20:36.989517 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260927 11:20:36.989691 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:36.979561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46583 (local address 127.0.98.62:41253)
0927 11:20:36.979614 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:36.979616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:36.979847 (+ 231us) server_negotiation.cc:407] Connection header received
0927 11:20:36.980567 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:36.980568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:36.980584 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:36.980650 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:36.981994 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.982807 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.983817 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:36.984214 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:36.984290 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:36.986497 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:36.986498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:36.986499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:36.986507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:36.988238 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:36.988625 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:36.988626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:36.988627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:36.988693 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:36.988930 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:36.988931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:36.988931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:36.989107 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:36.989188 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:36.989443 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 11:20:36.989544 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:20:36.990188 14985 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41253
I20260927 11:20:36.990302 14985 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:36.990509 14985 heartbeater.cc:511] Master 127.0.98.62:41253 requested a full tablet report, sending...
I20260927 11:20:36.990953 14659 ts_manager.cc:194] Registered new tserver with Master: bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513)
I20260927 11:20:36.991343 14659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46583
W20260927 11:20:37.066998 14988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:37.067224 14988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:37.067253 14988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:37.068411 14988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:37.068455 14988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:37.068482 14988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:20:37.070043 14988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44179
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:37.070299 14988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:37.070539 14988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:37.075757 14994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:37.075817 14996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:37.075788 14993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:37.075944 14988 server_base.cc:1034] running on GCE node
I20260927 11:20:37.076272 14988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:37.076521 14988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:37.077693 14988 hybrid_clock.cc:648] HybridClock initialized: now 1790508037077675 us; error 31 us; skew 500 ppm
Sep 27 11:20:37 dist-test-slave-lsvg krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:37.080210 14988 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:20:37.080633 14988 webserver.cc:460] Webserver started at http://127.0.98.3:44179/ using document root <none> and password file <none>
I20260927 11:20:37.080824 14988 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:37.080878 14988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:37.082316 14988 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:37.082847 15003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:37.083020 14988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:20:37.083091 14988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "1182b7f6a0c04d77a45cb23f5dd6117e"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "70eeaba52ce6b49fcfd5beee83889abd"
server_key_iv: "037db0f5c840c864d48ed8f044871ac1"
server_key_version: "encryptionkey@0"
I20260927 11:20:37.083421 14988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:37.092839 14988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:37.094834 14988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:37.094947 14988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:37.095131 14988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:37.095376 14988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:37.095404 14988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:37.095427 14988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:37.095443 14988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:37.101361 14988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37195
I20260927 11:20:37.101423 15116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37195 every 8 connection(s)
I20260927 11:20:37.101765 14988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:20:37.105610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 14988
Sep 27 11:20:37 dist-test-slave-lsvg krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:37.112013 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.102548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46291 (local address 127.0.98.62:41253)
0927 11:20:37.102620 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:37.102622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:37.102841 (+ 219us) server_negotiation.cc:407] Connection header received
0927 11:20:37.103311 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:37.103312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:37.103323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:37.103382 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:37.104305 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.104782 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.105877 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.106234 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.106303 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:37.108758 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:37.108759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:37.108760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:37.108766 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:37.110359 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:37.110863 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:37.110864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:37.110864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:37.110916 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:37.111224 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:37.111225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:37.111226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:37.111377 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:37.111442 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:37.111794 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 11:20:37.111883 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 11:20:37.112205 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.102535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:46291)
0927 11:20:37.102708 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.102759 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:37.103228 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:37.103436 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:37.103439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:37.104027 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:37.104221 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:37.104226 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.104866 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.104867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:37.105775 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:37.105778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.106315 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.106316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:37.106453 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:37.106968 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:37.106979 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:37.108671 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:37.110437 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:37.110444 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:37.110447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:37.110783 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:37.110991 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:37.110992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:37.110992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:37.111135 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:37.111446 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:37.111453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:37.111629 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 11:20:37.111770 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 11:20:37.111944 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260927 11:20:37.112720 15117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41253
I20260927 11:20:37.112848 15117 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:37.113029 15117 heartbeater.cc:511] Master 127.0.98.62:41253 requested a full tablet report, sending...
I20260927 11:20:37.113288 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.106051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.0.98.62:41253)
0927 11:20:37.106321 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:37.106323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:37.106330 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:20:37.106350 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:37.106350 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:37.106361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:37.106419 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:37.107158 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.107623 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.108558 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.108851 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.108929 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:37.110203 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:37.110205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:37.110205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:37.110212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:37.111812 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:37.112436 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:37.112437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:37.112437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:37.112501 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:37.112725 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:37.112726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:37.112727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:37.112862 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:37.113031 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:37.113093 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 11:20:37.113184 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":51,"threads_started":1}
I20260927 11:20:37.113582 14658 ts_manager.cc:194] Registered new tserver with Master: 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195)
I20260927 11:20:37.114028 14658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46291
I20260927 11:20:37.114164 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:20:37.117146 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.114586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.0.98.62:41253)
0927 11:20:37.114643 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:37.114645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:37.114677 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:20:37.114757 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:37.114758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:37.114794 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:37.114890 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:20:37.114989 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.115458 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.116319 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.116530 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.116604 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:37.116784 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:20:37.116923 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:20:37.117016 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:37.117036 (+ 20us) server_negotiation.cc:299] Negotiation successful
0927 11:20:37.117057 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 11:20:37.117431 14658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:20:37.118986 14658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:20:37.121192 14781 tablet_service.cc:1467] Processing CreateTablet for tablet e98a4e5d46614316b681fb6d725603a8 (DEFAULT_TABLE table=test-table [id=08f88cfe34f642bc837135e2f1ae2293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:20:37.121389 14781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98a4e5d46614316b681fb6d725603a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:37.122676 14739 tablet_bootstrap.cc:492] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70: Bootstrap starting.
I20260927 11:20:37.123076 14739 tablet_bootstrap.cc:654] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:37.123632 14739 tablet_bootstrap.cc:492] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70: No bootstrap required, opened a new log
I20260927 11:20:37.123711 14739 ts_tablet_manager.cc:1397] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 11:20:37.123836 14739 raft_consensus.cc:348] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:37.123903 14739 raft_consensus.cc:374] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:37.123948 14739 raft_consensus.cc:729] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2375972ca6477f95198463edc4ff70, State: Initialized, Role: FOLLOWER
I20260927 11:20:37.123998 14739 consensus_queue.cc:260] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:37.124084 14739 ts_tablet_manager.cc:1428] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:37.125304 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.120967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37195 (local address 127.0.0.1:46900)
0927 11:20:37.121005 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.121016 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:37.121072 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:37.121552 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:37.121554 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:37.121556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:37.121599 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:37.121601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.122977 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.122979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:37.124069 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:37.124072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.125040 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.125041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:37.125125 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:37.125129 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:20:37.125180 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 11:20:37.125231 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 11:20:37.125332 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.121111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46900 (local address 127.0.98.3:37195)
0927 11:20:37.121389 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:37.121394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:37.121405 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:20:37.121447 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:37.121449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:37.121466 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:37.121555 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:37.121657 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.122754 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.124200 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.124958 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.125040 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:37.125184 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:37.125213 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 11:20:37.125253 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260927 11:20:37.125876 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.121211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37513 (local address 127.0.0.1:47722)
0927 11:20:37.121346 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.121357 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:37.121417 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:37.121683 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:37.121684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:37.121686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:37.121726 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:37.121728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.123806 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.123807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:37.124886 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:37.124889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.125638 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.125640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:37.125700 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:37.125704 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:20:37.125764 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 11:20:37.125803 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260927 11:20:37.125893 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.121212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.0.98.2:37513)
0927 11:20:37.121409 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:37.121413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:37.121425 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:20:37.121499 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:37.121501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:37.121518 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:37.121599 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:37.121818 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.123663 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.124961 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.125534 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.125634 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:37.125700 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:37.125777 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 11:20:37.125814 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260927 11:20:37.126187 15051 tablet_service.cc:1467] Processing CreateTablet for tablet e98a4e5d46614316b681fb6d725603a8 (DEFAULT_TABLE table=test-table [id=08f88cfe34f642bc837135e2f1ae2293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:20:37.126593 15051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98a4e5d46614316b681fb6d725603a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:37.126616 14919 tablet_service.cc:1467] Processing CreateTablet for tablet e98a4e5d46614316b681fb6d725603a8 (DEFAULT_TABLE table=test-table [id=08f88cfe34f642bc837135e2f1ae2293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:20:37.126950 14919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98a4e5d46614316b681fb6d725603a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:37.130373 15127 tablet_bootstrap.cc:492] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e: Bootstrap starting.
I20260927 11:20:37.130673 15128 tablet_bootstrap.cc:492] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: Bootstrap starting.
I20260927 11:20:37.131129 15127 tablet_bootstrap.cc:654] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:37.131392 15128 tablet_bootstrap.cc:654] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:37.131788 15127 log.cc:824] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:37.132054 15128 log.cc:824] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:37.132939 15128 tablet_bootstrap.cc:492] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: No bootstrap required, opened a new log
I20260927 11:20:37.132946 15127 tablet_bootstrap.cc:492] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e: No bootstrap required, opened a new log
I20260927 11:20:37.133013 15127 ts_tablet_manager.cc:1397] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 11:20:37.133116 15128 ts_tablet_manager.cc:1397] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:20:37.135054 15127 raft_consensus.cc:348] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:37.135054 15128 raft_consensus.cc:348] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:37.135249 15128 raft_consensus.cc:374] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:37.135249 15127 raft_consensus.cc:374] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:37.135288 15128 raft_consensus.cc:729] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd6ce8a0153f473db53476f14a5b4544, State: Initialized, Role: FOLLOWER
I20260927 11:20:37.135288 15127 raft_consensus.cc:729] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1182b7f6a0c04d77a45cb23f5dd6117e, State: Initialized, Role: FOLLOWER
I20260927 11:20:37.135412 15128 consensus_queue.cc:260] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:37.135418 15127 consensus_queue.cc:260] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:37.135699 15128 ts_tablet_manager.cc:1428] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 11:20:37.135699 15127 ts_tablet_manager.cc:1428] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 11:20:37.135717 14985 heartbeater.cc:503] Master 127.0.98.62:41253 was elected leader, sending a full tablet report...
I20260927 11:20:37.136071 15117 heartbeater.cc:503] Master 127.0.98.62:41253 was elected leader, sending a full tablet report...
W20260927 11:20:37.229405 14986 tablet.cc:2369] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:20:37.272050 15132 raft_consensus.cc:482] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:20:37.272169 15132 raft_consensus.cc:504] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:37.272502 15132 leader_election.cc:288] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513), 3e2375972ca6477f95198463edc4ff70 (127.0.98.1:42543)
I20260927 11:20:37.275336 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.272802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51087 (local address 127.0.98.2:37513)
0927 11:20:37.272847 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:37.272849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:37.272859 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:20:37.272955 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:37.272956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:37.272973 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:37.273040 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:37.273217 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.273686 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.274596 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.274983 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.275068 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:37.275079 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:37.275230 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 11:20:37.275266 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 11:20:37.275337 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.272693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37513 (local address 127.0.98.3:51087)
0927 11:20:37.272794 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.272807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:37.272876 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:37.273063 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:37.273064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:37.273065 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:37.273142 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:37.273143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.273771 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.273772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:37.274500 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:37.274502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.275067 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.275068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:37.275153 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:37.275156 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:20:37.275225 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 11:20:37.275265 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":1}
I20260927 11:20:37.275472 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.272693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42543 (local address 127.0.98.3:35287)
0927 11:20:37.272979 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.272993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:37.273085 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:37.273241 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:37.273242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:37.273243 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:37.273321 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:37.273323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.273938 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.273939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:37.274659 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:37.274661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.275213 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.275214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:37.275287 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:37.275290 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:20:37.275355 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 11:20:37.275394 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":108,"thread_start_us":121,"threads_started":1}
I20260927 11:20:37.275460 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.272775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35287 (local address 127.0.98.1:42543)
0927 11:20:37.272821 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:37.272824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:37.273079 (+ 255us) server_negotiation.cc:407] Connection header received
0927 11:20:37.273144 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:37.273145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:37.273157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:37.273226 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:37.273399 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.273855 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.274749 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.275128 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.275210 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:37.275219 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:37.275359 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 11:20:37.275394 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:20:37.275763 14801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98a4e5d46614316b681fb6d725603a8" candidate_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2375972ca6477f95198463edc4ff70" is_pre_election: true
I20260927 11:20:37.275825 14939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98a4e5d46614316b681fb6d725603a8" candidate_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6ce8a0153f473db53476f14a5b4544" is_pre_election: true
I20260927 11:20:37.275928 14801 raft_consensus.cc:2454] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1182b7f6a0c04d77a45cb23f5dd6117e in term 0.
I20260927 11:20:37.275951 14939 raft_consensus.cc:2454] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1182b7f6a0c04d77a45cb23f5dd6117e in term 0.
I20260927 11:20:37.276154 15007 leader_election.cc:302] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1182b7f6a0c04d77a45cb23f5dd6117e, 3e2375972ca6477f95198463edc4ff70; no voters:
I20260927 11:20:37.276324 15132 raft_consensus.cc:2781] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:20:37.276408 15132 raft_consensus.cc:482] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:20:37.276443 15132 raft_consensus.cc:3037] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:37.277253 15132 raft_consensus.cc:504] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:37.277403 15132 leader_election.cc:288] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [CANDIDATE]: Term 1 election: Requested vote from peers bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513), 3e2375972ca6477f95198463edc4ff70 (127.0.98.1:42543)
I20260927 11:20:37.277570 14939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98a4e5d46614316b681fb6d725603a8" candidate_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd6ce8a0153f473db53476f14a5b4544"
I20260927 11:20:37.277572 14801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98a4e5d46614316b681fb6d725603a8" candidate_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e2375972ca6477f95198463edc4ff70"
I20260927 11:20:37.277649 14939 raft_consensus.cc:3037] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:37.277678 14801 raft_consensus.cc:3037] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:37.278373 14939 raft_consensus.cc:2454] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1182b7f6a0c04d77a45cb23f5dd6117e in term 1.
I20260927 11:20:37.278542 15004 leader_election.cc:302] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1182b7f6a0c04d77a45cb23f5dd6117e, bd6ce8a0153f473db53476f14a5b4544; no voters:
I20260927 11:20:37.278645 15132 raft_consensus.cc:2781] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:20:37.278796 14801 raft_consensus.cc:2454] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1182b7f6a0c04d77a45cb23f5dd6117e in term 1.
I20260927 11:20:37.278838 15132 raft_consensus.cc:686] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 1 LEADER]: Becoming Leader. State: Replica: 1182b7f6a0c04d77a45cb23f5dd6117e, State: Running, Role: LEADER
I20260927 11:20:37.278937 15132 consensus_queue.cc:237] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:37.279930 14658 catalog_manager.cc:5495] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } health_report { overall_health: UNKNOWN } } }
I20260927 11:20:37.310259 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.308049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.0.98.3:37195)
0927 11:20:37.308112 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:37.308114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:37.308124 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:20:37.308166 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:37.308167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:37.308187 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:37.308257 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:20:37.308372 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.308894 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.309599 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:37.309786 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:37.309842 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:37.309965 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:20:37.310064 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:20:37.310125 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:37.310157 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 11:20:37.310174 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:20:37.312132 14939 raft_consensus.cc:1261] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544 [term 1 FOLLOWER]: Refusing update from remote peer 1182b7f6a0c04d77a45cb23f5dd6117e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:20:37.312219 14801 raft_consensus.cc:1261] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Refusing update from remote peer 1182b7f6a0c04d77a45cb23f5dd6117e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:20:37.312572 15134 consensus_queue.cc:1035] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:20:37.312724 15132 consensus_queue.cc:1035] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:20:37.314838 15137 mvcc.cc:205] Tried to move back new op lower bound from 7333920920827539456 to 7333920920695779328. Current Snapshot: MvccSnapshot[applied={T|T < 7333920920827539456}]
I20260927 11:20:37.315997 15138 mvcc.cc:205] Tried to move back new op lower bound from 7333920920827539456 to 7333920920695779328. Current Snapshot: MvccSnapshot[applied={T|T < 7333920920827539456}]
I20260927 11:20:37.317234 14658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:20:37.317348 14658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:20:37.318114 14658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Sending DeleteTablet for 3 replicas of tablet e98a4e5d46614316b681fb6d725603a8
I20260927 11:20:37.318427 14781 tablet_service.cc:1514] Processing DeleteTablet for tablet e98a4e5d46614316b681fb6d725603a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59174
I20260927 11:20:37.318519 14856 tablet_replica.cc:330] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70: stopping tablet replica
I20260927 11:20:37.318567 14856 raft_consensus.cc:2229] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:20:37.318552 15051 tablet_service.cc:1514] Processing DeleteTablet for tablet e98a4e5d46614316b681fb6d725603a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46900
I20260927 11:20:37.318607 14856 raft_consensus.cc:2258] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:20:37.318645 14919 tablet_service.cc:1514] Processing DeleteTablet for tablet e98a4e5d46614316b681fb6d725603a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47722
I20260927 11:20:37.318864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 14698
I20260927 11:20:37.318953 14856 ts_tablet_manager.cc:1905] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:20:37.319490 15146 tablet_replica.cc:330] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e: stopping tablet replica
I20260927 11:20:37.319590 15146 raft_consensus.cc:2229] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:20:37.319702 15146 raft_consensus.cc:2258] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:20:37.320111 15146 ts_tablet_manager.cc:1905] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:20:37.321751 15146 ts_tablet_manager.cc:1918] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:20:37.321846 15146 log.cc:1196] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/e98a4e5d46614316b681fb6d725603a8
I20260927 11:20:37.322113 15146 ts_tablet_manager.cc:1939] T e98a4e5d46614316b681fb6d725603a8 P 1182b7f6a0c04d77a45cb23f5dd6117e: Deleting consensus metadata
I20260927 11:20:37.322508 14643 catalog_manager.cc:4841] TS 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195): tablet e98a4e5d46614316b681fb6d725603a8 (table test-table [id=08f88cfe34f642bc837135e2f1ae2293]) successfully deleted
I20260927 11:20:37.325210 15147 tablet_replica.cc:330] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: stopping tablet replica
I20260927 11:20:37.325351 15147 raft_consensus.cc:2229] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:20:37.325466 15147 raft_consensus.cc:2258] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:20:37.325881 15147 ts_tablet_manager.cc:1905] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 11:20:37.326538 14646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42543 (error 108)
I20260927 11:20:37.326835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 14850
I20260927 11:20:37.327165 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.327046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42543 (local address 127.0.0.1:59180)
0927 11:20:37.327098 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.327131 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 11:20:37.327373 14646 catalog_manager.cc:4584] TS 3e2375972ca6477f95198463edc4ff70 (127.0.98.1:42543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e98a4e5d46614316b681fb6d725603a8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42543: connect: Connection refused (error 111)
I20260927 11:20:37.334288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 14988
I20260927 11:20:37.334405 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.334253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37513 (local address 127.0.0.1:47732)
0927 11:20:37.334333 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.334369 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 11:20:37.339329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 14626
I20260927 11:20:37.344584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41253
--webserver_interface=127.0.98.62
--webserver_port=46641
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:37.425777 15149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:37.426040 15149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:37.426079 15149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:37.427266 15149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:20:37.427341 15149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:37.427366 15149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:37.427386 15149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:20:37.427415 15149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:20:37.428942 15149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:37.429255 15149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:37.429489 15149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:37.434497 15154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:37.434562 15155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:37.434538 15157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:37.434571 15149 server_base.cc:1034] running on GCE node
I20260927 11:20:37.434969 15149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:37.435369 15149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:37.436542 15149 hybrid_clock.cc:648] HybridClock initialized: now 1790508037436522 us; error 29 us; skew 500 ppm
Sep 27 11:20:37 dist-test-slave-lsvg krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:37.439131 15149 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:20:37.439595 15149 webserver.cc:460] Webserver started at http://127.0.98.62:46641/ using document root <none> and password file <none>
I20260927 11:20:37.439850 15149 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:37.439922 15149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:37.441393 15149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:37.442090 15164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:37.442307 15149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:37.442394 15149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "6e37444cc864412daa8ac39286643c24"
format_stamp: "Formatted at 2026-09-27 11:20:35 on dist-test-slave-lsvg"
server_key: "11c7f4d12d2324b4232ffd3dc80ff249"
server_key_iv: "e4a057045ae891958472d7fb0754ca19"
server_key_version: "encryptionkey@0"
I20260927 11:20:37.442737 15149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:37.455020 15149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:37.457453 15149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:37.457572 15149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:37.461567 15149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41253
I20260927 11:20:37.461601 15216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41253 every 8 connection(s)
I20260927 11:20:37.461987 15149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:20:37.463578 15217 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:20:37.464268 15217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Bootstrap starting.
I20260927 11:20:37.469396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 15149
I20260927 11:20:37.469691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42543
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41253
--webserver_port=42449
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:20:37.472579 15217 log.cc:824] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:37.474911 15217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:20:37.475222 15217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24: Bootstrap complete.
I20260927 11:20:37.477507 15217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260927 11:20:37.477748 15217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e37444cc864412daa8ac39286643c24, State: Initialized, Role: FOLLOWER
I20260927 11:20:37.477926 15217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260927 11:20:37.478014 15217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:20:37.478084 15217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:20:37.478144 15217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 2 FOLLOWER]: Advancing to term 3
I20260927 11:20:37.479094 15217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260927 11:20:37.479216 15217 leader_election.cc:302] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e37444cc864412daa8ac39286643c24; no voters:
I20260927 11:20:37.479423 15217 leader_election.cc:288] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 11:20:37.479455 15221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 11:20:37.479563 15221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [term 3 LEADER]: Becoming Leader. State: Replica: 6e37444cc864412daa8ac39286643c24, State: Running, Role: LEADER
I20260927 11:20:37.479653 15221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } }
I20260927 11:20:37.479686 15217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:20:37.479911 15222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6e37444cc864412daa8ac39286643c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } } }
I20260927 11:20:37.479910 15221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e37444cc864412daa8ac39286643c24. Latest consensus state: current_term: 3 leader_uuid: "6e37444cc864412daa8ac39286643c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e37444cc864412daa8ac39286643c24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41253 } } }
I20260927 11:20:37.480098 15222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:37.480123 15221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e37444cc864412daa8ac39286643c24 [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:37.480698 15230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:20:37.481500 15230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:20:37.481624 15230 catalog_manager.cc:1259] Loaded cluster ID: 8320e2d00d484d08b68195d0a2d039eb
I20260927 11:20:37.481695 15230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 11:20:37.482079 15230 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f81ecd8a9ad google::LogMessage::Fail() at ??:0
@ 0x7f81ecd8d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f81ecd8a4de google::LogMessage::Flush() at ??:0
@ 0x7f81ecd8ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f81f4cab0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f81f4cb48b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f81eec580d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f81eec4d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f81efab16db start_thread at ??:0
@ 0x7f81eb88861f clone at ??:0
W20260927 11:20:37.565523 15219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:37.565730 15219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:37.565757 15219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:37.567054 15219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:37.567111 15219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:37.567140 15219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:20:37.568626 15219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42449
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:37.568884 15219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:37.569166 15219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:37.575670 15254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:37.575708 15219 server_base.cc:1034] running on GCE node
W20260927 11:20:37.575688 15253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:37.575922 15256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:37.576192 15219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:37.576488 15219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:37.577672 15219 hybrid_clock.cc:648] HybridClock initialized: now 1790508037577662 us; error 32 us; skew 500 ppm
Sep 27 11:20:37 dist-test-slave-lsvg krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:37.580698 15219 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:20:37.581254 15219 webserver.cc:460] Webserver started at http://127.0.98.1:42449/ using document root <none> and password file <none>
I20260927 11:20:37.581508 15219 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:37.581579 15219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:37.583000 15219 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:37.583806 15263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:37.583995 15219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:37.584079 15219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "3e2375972ca6477f95198463edc4ff70"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "f52887a039153d5233a6326d22a96234"
server_key_iv: "01a3c026c34101b5c91be23d3f65e269"
server_key_version: "encryptionkey@0"
I20260927 11:20:37.584612 15219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:37.602213 15219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:37.604640 15219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:37.604811 15219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:37.605096 15219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:37.605585 15270 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 11:20:37.607635 15219 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 11:20:37.607715 15219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 11:20:37.607769 15219 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 11:20:37.608461 15219 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 11:20:37.608518 15219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 11:20:37.608613 15270 tablet_bootstrap.cc:492] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70: Bootstrap starting.
I20260927 11:20:37.610634 15270 log.cc:824] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:37.613720 15270 tablet_bootstrap.cc:492] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:20:37.614058 15270 tablet_bootstrap.cc:492] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70: Bootstrap complete.
I20260927 11:20:37.614228 15270 ts_tablet_manager.cc:1397] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260927 11:20:37.615811 15219 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42543
I20260927 11:20:37.615844 15378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42543 every 8 connection(s)
I20260927 11:20:37.616185 15219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:20:37.616446 15270 raft_consensus.cc:348] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:37.616823 15270 raft_consensus.cc:729] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2375972ca6477f95198463edc4ff70, State: Initialized, Role: FOLLOWER
I20260927 11:20:37.616984 15270 consensus_queue.cc:260] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:37.617317 15270 ts_tablet_manager.cc:1428] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 11:20:37.625160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 15219
I20260927 11:20:37.625447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37513
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41253
--webserver_port=37049
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:37.686029 15381 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:20:37.616925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:38723)
0927 11:20:37.617192 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.617240 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:37.617415 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:37.685805 (+ 68390us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":104,"threads_started":1}
W20260927 11:20:37.686293 15379 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41253 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41253: Client connection negotiation failed: client connection to 127.0.98.62:41253: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 11:20:37.686678 15381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.686543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:43567)
0927 11:20:37.686600 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.686642 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 11:20:37.686987 15381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.686899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:43665)
0927 11:20:37.686944 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.686966 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 11:20:37.687110 15379 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 11:20:37.707962 15383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:37.708165 15383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:37.708189 15383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:37.709420 15383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:37.709461 15383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:37.709511 15383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:20:37.711045 15383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37049
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:37.711292 15383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:37.711522 15383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:37.716493 15389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:37.716514 15391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:37.716504 15388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:37.716843 15383 server_base.cc:1034] running on GCE node
I20260927 11:20:37.717097 15383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:37.717353 15383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:37.718509 15383 hybrid_clock.cc:648] HybridClock initialized: now 1790508037718496 us; error 26 us; skew 500 ppm
Sep 27 11:20:37 dist-test-slave-lsvg krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:37.721098 15383 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:20:37.721555 15383 webserver.cc:460] Webserver started at http://127.0.98.2:37049/ using document root <none> and password file <none>
I20260927 11:20:37.721800 15383 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:37.721870 15383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:37.723210 15383 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:37.723959 15398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:37.724179 15383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:37.724264 15383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "bd6ce8a0153f473db53476f14a5b4544"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "21fb9d47694ac632de50781fb0ec5ed8"
server_key_iv: "05c03c18cb7d820c20bcb564f87ba2f7"
server_key_version: "encryptionkey@0"
I20260927 11:20:37.724596 15383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:37.751917 15383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:37.754172 15383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:37.754307 15383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:37.754508 15383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:37.754987 15405 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 11:20:37.756461 15405 ts_tablet_manager.cc:1852] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 11:20:37.756773 15405 ts_tablet_manager.cc:1905] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:20:37.759196 15405 ts_tablet_manager.cc:1918] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:20:37.759312 15405 log.cc:1196] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/e98a4e5d46614316b681fb6d725603a8
I20260927 11:20:37.759595 15405 ts_tablet_manager.cc:1939] T e98a4e5d46614316b681fb6d725603a8 P bd6ce8a0153f473db53476f14a5b4544: Deleting consensus metadata
I20260927 11:20:37.759830 15383 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 11:20:37.759879 15383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 11:20:37.759907 15383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:37.759920 15383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:37.766140 15383 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37513
I20260927 11:20:37.766188 15512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37513 every 8 connection(s)
I20260927 11:20:37.766513 15383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:20:37.767524 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.767145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:41415)
0927 11:20:37.767316 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.767387 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
W20260927 11:20:37.767751 15513 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41253 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41253: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
I20260927 11:20:37.768033 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.767949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:45821)
0927 11:20:37.767993 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.768018 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 11:20:37.768297 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.768223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:59043)
0927 11:20:37.768260 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.768282 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 11:20:37.768393 15513 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 11:20:37.771166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 15383
I20260927 11:20:37.771430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37195
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41253
--webserver_port=44179
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:37.851754 15516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:37.851971 15516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:37.851996 15516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:37.853231 15516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:37.853276 15516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:37.853303 15516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:20:37.854821 15516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44179
--tserver_master_addrs=127.0.98.62:41253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:37.855118 15516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:37.855391 15516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:37.860492 15521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:37.860551 15522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:37.860491 15524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:37.860764 15516 server_base.cc:1034] running on GCE node
I20260927 11:20:37.860999 15516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:37.861267 15516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:37.862408 15516 hybrid_clock.cc:648] HybridClock initialized: now 1790508037862398 us; error 27 us; skew 500 ppm
Sep 27 11:20:37 dist-test-slave-lsvg krb5kdc[14079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:37.864879 15516 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:20:37.865424 15516 webserver.cc:460] Webserver started at http://127.0.98.3:44179/ using document root <none> and password file <none>
I20260927 11:20:37.865669 15516 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:37.865739 15516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:37.867061 15516 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:20:37.867765 15531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:37.867982 15516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:20:37.868074 15516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "1182b7f6a0c04d77a45cb23f5dd6117e"
format_stamp: "Formatted at 2026-09-27 11:20:36 on dist-test-slave-lsvg"
server_key: "70eeaba52ce6b49fcfd5beee83889abd"
server_key_iv: "037db0f5c840c864d48ed8f044871ac1"
server_key_version: "encryptionkey@0"
I20260927 11:20:37.868405 15516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:37.886634 15516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:37.889242 15516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:37.889364 15516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:37.889549 15516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:37.889863 15516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:37.889899 15516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:37.889953 15516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:37.889981 15516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:37.895507 15516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37195
I20260927 11:20:37.895591 15644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37195 every 8 connection(s)
I20260927 11:20:37.895931 15516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:20:37.896306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 15516
I20260927 11:20:37.897221 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.896668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:53267)
0927 11:20:37.896950 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.897007 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":111,"threads_started":1}
W20260927 11:20:37.897642 15645 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41253 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41253: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
I20260927 11:20:37.897962 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.897839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:51037)
0927 11:20:37.897909 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.897940 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 11:20:37.898272 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:37.898209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:35031)
0927 11:20:37.898237 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:37.898257 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260927 11:20:37.898391 15645 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 11:20:38.687788 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:38.687549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:41247)
0927 11:20:38.687677 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:38.687722 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260927 11:20:38.768904 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:38.768719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:38245)
0927 11:20:38.768833 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:38.768870 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260927 11:20:38.898940 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:38.898737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:38647)
0927 11:20:38.898853 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:38.898895 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260927 11:20:39.056632 15652 raft_consensus.cc:482] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:20:39.056772 15652 raft_consensus.cc:504] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:39.057188 15652 leader_election.cc:288] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195), bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513)
Sep 27 11:20:39 dist-test-slave-lsvg krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 11:20:39 dist-test-slave-lsvg krb5kdc[14079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 11:20:39.068354 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:39.057351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37195 (local address 127.0.98.1:44635)
0927 11:20:39.057406 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:39.057418 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:39.057477 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:39.058442 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:39.058445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:39.058847 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:39.059088 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:39.059094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:39.060363 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:39.060364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:39.061331 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:39.061333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:39.061953 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:39.061955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:39.062061 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:39.062678 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:39.062687 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:39.064641 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:39.066667 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:39.066668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:39.066669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:39.066918 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:39.067295 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:39.067297 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:39.067297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:39.067369 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:39.067867 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:39.067873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:39.067993 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 11:20:39.068089 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 11:20:39.068258 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":71}
I20260927 11:20:39.068356 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:39.057479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44635 (local address 127.0.98.3:37195)
0927 11:20:39.057612 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:39.057622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:39.057676 (+ 54us) server_negotiation.cc:407] Connection header received
0927 11:20:39.058181 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:39.058184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:39.058231 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:39.058364 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:39.059249 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:39.060218 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:39.061438 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:39.061852 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:39.061985 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:39.064768 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:39.064772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:39.064772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:39.064784 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:39.066580 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:39.067021 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:39.067024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:39.067024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:39.067109 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:39.067443 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:39.067444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:39.067444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:39.067766 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:39.067871 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:39.068141 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 11:20:39.068278 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":52,"threads_started":1}
I20260927 11:20:39.068811 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:39.057481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37513 (local address 127.0.98.1:53681)
0927 11:20:39.057564 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:39.057574 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:39.057612 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:39.058702 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:39.058703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:39.058882 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:39.059314 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:39.059316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:39.060440 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:39.060441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:39.061330 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:39.061333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:39.061933 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:39.061934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:39.062054 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:39.062738 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:39.062743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:39.064311 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:39.066515 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:39.066523 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:39.066526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:39.066892 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:39.067223 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:39.067224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:39.067225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:39.067361 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:39.068483 (+ 1122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:39.068484 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:39.068540 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 11:20:39.068641 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 11:20:39.068729 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":39,"thread_start_us":52,"threads_started":1}
I20260927 11:20:39.068959 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:39.057694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53681 (local address 127.0.98.2:37513)
0927 11:20:39.057852 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:39.057863 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:39.057887 (+ 24us) server_negotiation.cc:407] Connection header received
0927 11:20:39.058438 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:39.058441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:39.058490 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:39.058633 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:39.059452 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:39.060323 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:39.061431 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:39.061844 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:39.061954 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:39.064463 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:39.064468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:39.064469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:39.064481 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:39.066415 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:39.067001 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:39.067005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:39.067005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:39.067104 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:39.067441 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:39.067442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:39.067443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:39.067772 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:39.067859 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:39.068718 (+ 859us) server_negotiation.cc:299] Negotiation successful
0927 11:20:39.068858 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260927 11:20:39.069386 15264 leader_election.cc:341] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195): Not found: Tablet not found: e98a4e5d46614316b681fb6d725603a8
W20260927 11:20:39.069721 15264 leader_election.cc:341] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513): Not found: Tablet not found: e98a4e5d46614316b681fb6d725603a8
I20260927 11:20:39.069772 15264 leader_election.cc:302] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e2375972ca6477f95198463edc4ff70; no voters: 1182b7f6a0c04d77a45cb23f5dd6117e, bd6ce8a0153f473db53476f14a5b4544
I20260927 11:20:39.069916 15652 raft_consensus.cc:2726] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:20:39.688565 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:39.688299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:50419)
0927 11:20:39.688464 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:39.688514 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20260927 11:20:39.769682 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:39.769490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:51817)
0927 11:20:39.769608 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:39.769644 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260927 11:20:39.899631 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:39.899425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:36247)
0927 11:20:39.899545 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:39.899589 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260927 11:20:40.615360 15659 raft_consensus.cc:482] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:20:40.615464 15659 raft_consensus.cc:504] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:40.615773 15659 leader_election.cc:288] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195), bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513)
W20260927 11:20:40.616240 15264 leader_election.cc:341] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195): Not found: Tablet not found: e98a4e5d46614316b681fb6d725603a8
W20260927 11:20:40.616323 15264 leader_election.cc:341] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513): Not found: Tablet not found: e98a4e5d46614316b681fb6d725603a8
I20260927 11:20:40.616339 15264 leader_election.cc:302] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e2375972ca6477f95198463edc4ff70; no voters: 1182b7f6a0c04d77a45cb23f5dd6117e, bd6ce8a0153f473db53476f14a5b4544
I20260927 11:20:40.616415 15659 raft_consensus.cc:2726] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:20:40.689460 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:40.689214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:40577)
0927 11:20:40.689354 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:40.689395 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260927 11:20:40.770324 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:40.770127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:41519)
0927 11:20:40.770252 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:40.770287 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260927 11:20:40.900368 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:40.900128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:42573)
0927 11:20:40.900285 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:40.900325 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260927 11:20:41.690240 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:41.690006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:54909)
0927 11:20:41.690146 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:41.690190 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260927 11:20:41.770973 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:41.770770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:60951)
0927 11:20:41.770893 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:41.770932 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260927 11:20:41.901130 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:41.900863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:52999)
0927 11:20:41.901001 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:41.901040 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260927 11:20:42.120951 15666 raft_consensus.cc:482] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:20:42.121110 15666 raft_consensus.cc:504] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:42.121284 15666 leader_election.cc:288] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195), bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513)
W20260927 11:20:42.121692 15264 leader_election.cc:341] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195): Not found: Tablet not found: e98a4e5d46614316b681fb6d725603a8
W20260927 11:20:42.121804 15264 leader_election.cc:341] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513): Not found: Tablet not found: e98a4e5d46614316b681fb6d725603a8
I20260927 11:20:42.121834 15264 leader_election.cc:302] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e2375972ca6477f95198463edc4ff70; no voters: 1182b7f6a0c04d77a45cb23f5dd6117e, bd6ce8a0153f473db53476f14a5b4544
I20260927 11:20:42.121925 15666 raft_consensus.cc:2726] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:20:42.691051 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:42.690760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:59739)
0927 11:20:42.690945 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:42.690996 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
I20260927 11:20:42.771827 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:42.771490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:52161)
0927 11:20:42.771672 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:42.771761 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":102,"threads_started":1}
I20260927 11:20:42.901870 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:42.901591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:60341)
0927 11:20:42.901772 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:42.901811 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260927 11:20:43.691815 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:43.691578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:60113)
0927 11:20:43.691728 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:43.691768 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260927 11:20:43.772545 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:43.772324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:56515)
0927 11:20:43.772460 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:43.772503 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260927 11:20:43.902590 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:43.902362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:42359)
0927 11:20:43.902489 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:43.902537 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260927 11:20:44.692497 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:44.692286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:46611)
0927 11:20:44.692409 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:44.692450 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260927 11:20:44.773353 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:44.773134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:44819)
0927 11:20:44.773266 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:44.773308 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260927 11:20:44.815196 15675 raft_consensus.cc:482] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:20:44.815416 15675 raft_consensus.cc:504] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:44.815627 15675 leader_election.cc:288] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195), bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513)
W20260927 11:20:44.816098 15264 leader_election.cc:341] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195): Not found: Tablet not found: e98a4e5d46614316b681fb6d725603a8
W20260927 11:20:44.816165 15264 leader_election.cc:341] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513): Not found: Tablet not found: e98a4e5d46614316b681fb6d725603a8
I20260927 11:20:44.816182 15264 leader_election.cc:302] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e2375972ca6477f95198463edc4ff70; no voters: 1182b7f6a0c04d77a45cb23f5dd6117e, bd6ce8a0153f473db53476f14a5b4544
I20260927 11:20:44.816293 15675 raft_consensus.cc:2726] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:20:44.903321 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:44.903086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:42041)
0927 11:20:44.903231 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:44.903268 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260927 11:20:45.693248 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:45.692993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:55665)
0927 11:20:45.693157 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:45.693199 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260927 11:20:45.774050 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:45.773828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:47107)
0927 11:20:45.773959 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:45.774001 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260927 11:20:45.904199 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:45.903898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:34545)
0927 11:20:45.904096 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:45.904152 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260927 11:20:46.694063 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:46.693826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:37291)
0927 11:20:46.693972 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:46.694015 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260927 11:20:46.774873 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:46.774636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:58103)
0927 11:20:46.774782 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:46.774824 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260927 11:20:46.904978 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:46.904756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:44247)
0927 11:20:46.904888 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:46.904932 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260927 11:20:47.694846 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:47.694591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:51895)
0927 11:20:47.694746 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:47.694795 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20260927 11:20:47.775553 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:47.775348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:43227)
0927 11:20:47.775473 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:47.775510 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260927 11:20:47.905860 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:47.905549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:43365)
0927 11:20:47.905765 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:47.905811 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":106,"threads_started":1}
I20260927 11:20:48.695672 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:48.695413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:32961)
0927 11:20:48.695578 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:48.695623 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":79,"threads_started":1}
I20260927 11:20:48.776242 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:48.775986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:46121)
0927 11:20:48.776159 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:48.776198 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260927 11:20:48.906654 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:48.906357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:48857)
0927 11:20:48.906519 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:48.906601 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
I20260927 11:20:49.696485 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:49.696240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:34935)
0927 11:20:49.696390 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:49.696438 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260927 11:20:49.776924 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:49.776711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:59981)
0927 11:20:49.776826 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:49.776879 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260927 11:20:49.907444 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:49.907224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:43671)
0927 11:20:49.907351 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:49.907393 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260927 11:20:50.697264 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:50.697015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:56693)
0927 11:20:50.697180 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:50.697218 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":109,"threads_started":1}
I20260927 11:20:50.777642 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:50.777439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:45293)
0927 11:20:50.777553 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:50.777594 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260927 11:20:50.908126 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:50.907930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:55261)
0927 11:20:50.908052 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:50.908083 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260927 11:20:51.698036 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:51.697743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:43835)
0927 11:20:51.697897 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:51.697976 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":61,"threads_started":1}
I20260927 11:20:51.778411 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:51.778156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:46311)
0927 11:20:51.778320 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:51.778363 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260927 11:20:51.908924 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:51.908666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:33981)
0927 11:20:51.908815 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:51.908876 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260927 11:20:51.911002 15698 raft_consensus.cc:482] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:20:51.911080 15698 raft_consensus.cc:504] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1182b7f6a0c04d77a45cb23f5dd6117e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37195 } } peers { permanent_uuid: "bd6ce8a0153f473db53476f14a5b4544" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37513 } } peers { permanent_uuid: "3e2375972ca6477f95198463edc4ff70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42543 } }
I20260927 11:20:51.911222 15698 leader_election.cc:288] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195), bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513)
W20260927 11:20:51.911576 15264 leader_election.cc:341] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1182b7f6a0c04d77a45cb23f5dd6117e (127.0.98.3:37195): Not found: Tablet not found: e98a4e5d46614316b681fb6d725603a8
W20260927 11:20:51.911651 15264 leader_election.cc:341] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd6ce8a0153f473db53476f14a5b4544 (127.0.98.2:37513): Not found: Tablet not found: e98a4e5d46614316b681fb6d725603a8
I20260927 11:20:51.911669 15264 leader_election.cc:302] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e2375972ca6477f95198463edc4ff70; no voters: 1182b7f6a0c04d77a45cb23f5dd6117e, bd6ce8a0153f473db53476f14a5b4544
I20260927 11:20:51.911774 15698 raft_consensus.cc:2726] T e98a4e5d46614316b681fb6d725603a8 P 3e2375972ca6477f95198463edc4ff70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:20:52.698999 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:52.698764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.1:52777)
0927 11:20:52.698903 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:52.698953 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260927 11:20:52.779163 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:52.778937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.2:44871)
0927 11:20:52.779080 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:52.779120 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260927 11:20:52.900681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 15219
I20260927 11:20:52.905649 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 15383
I20260927 11:20:52.909663 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:52.909466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41253 (local address 127.0.98.3:41125)
0927 11:20:52.909583 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:52.909619 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260927 11:20:52.910104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 15516
2026-09-27T11:20:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18417 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:20:52 dist-test-slave-lsvg krb5kdc[15704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:20:52 dist-test-slave-lsvg krb5kdc[15704](info): set up 2 sockets
Sep 27 11:20:52 dist-test-slave-lsvg krb5kdc[15704](info): commencing operation
krb5kdc: starting...
W20260927 11:20:53.957291 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:20:54 dist-test-slave-lsvg krb5kdc[15704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:20:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:20:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:20:54.081007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41273
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:54.161478 15717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:54.161689 15717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:54.161711 15717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:54.162895 15717 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 11:20:54.162940 15717 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 11:20:54.162952 15717 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 11:20:54.162964 15717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:20:54.162977 15717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:54.163019 15717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:54.163049 15717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:20:54.163065 15717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:20:54.164535 15717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:54.164783 15717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:54.165031 15717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:54.169900 15722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:54.169961 15717 server_base.cc:1034] running on GCE node
W20260927 11:20:54.169916 15723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:54.169934 15725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:54.170501 15717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:54.170753 15717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:54.171905 15717 hybrid_clock.cc:648] HybridClock initialized: now 1790508054171895 us; error 29 us; skew 500 ppm
Sep 27 11:20:54 dist-test-slave-lsvg krb5kdc[15704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:54.174525 15717 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:20:54.175019 15717 webserver.cc:460] Webserver started at http://127.0.98.62:43855/ using document root <none> and password file <none>
I20260927 11:20:54.175282 15717 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:54.175328 15717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:54.175413 15717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:20:54.176457 15717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "66bae122a7154fa58a97062b3d5db0a9"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "b04c2a4f3d8dd4f8f01d8f5bfb0e4021"
server_key_iv: "08508527e82c5380d9e997368799e843"
server_key_version: "encryptionkey@0"
I20260927 11:20:54.176759 15717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "66bae122a7154fa58a97062b3d5db0a9"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "b04c2a4f3d8dd4f8f01d8f5bfb0e4021"
server_key_iv: "08508527e82c5380d9e997368799e843"
server_key_version: "encryptionkey@0"
I20260927 11:20:54.178107 15717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:20:54.178877 15732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:54.179056 15717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:54.179128 15717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "66bae122a7154fa58a97062b3d5db0a9"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "b04c2a4f3d8dd4f8f01d8f5bfb0e4021"
server_key_iv: "08508527e82c5380d9e997368799e843"
server_key_version: "encryptionkey@0"
I20260927 11:20:54.179220 15717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:54.190953 15717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:54.193472 15717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:54.193619 15717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:54.198341 15717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41273
I20260927 11:20:54.198380 15784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41273 every 8 connection(s)
I20260927 11:20:54.198740 15717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:20:54.199630 15785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:54.202037 15785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Bootstrap starting.
I20260927 11:20:54.202620 15785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:54.202924 15785 log.cc:824] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:54.203678 15785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: No bootstrap required, opened a new log
I20260927 11:20:54.205322 15785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } }
I20260927 11:20:54.205483 15785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:54.205531 15785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bae122a7154fa58a97062b3d5db0a9, State: Initialized, Role: FOLLOWER
I20260927 11:20:54.205595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 15717
I20260927 11:20:54.205718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:20:54.205709 15785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } }
I20260927 11:20:54.205821 15785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:20:54.205886 15785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:20:54.205901 392 external_mini_cluster.cc:1428] Setting key 9a66006517a7fed2da37a571d1246a0b
I20260927 11:20:54.205938 15785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:54.206645 15785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } }
I20260927 11:20:54.206760 15785 leader_election.cc:302] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66bae122a7154fa58a97062b3d5db0a9; no voters:
I20260927 11:20:54.206954 15785 leader_election.cc:288] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:20:54.207024 15790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:20:54.207191 15790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 1 LEADER]: Becoming Leader. State: Replica: 66bae122a7154fa58a97062b3d5db0a9, State: Running, Role: LEADER
I20260927 11:20:54.207202 15785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:20:54.207326 15790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } }
I20260927 11:20:54.207753 15791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66bae122a7154fa58a97062b3d5db0a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } } }
I20260927 11:20:54.207739 15792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66bae122a7154fa58a97062b3d5db0a9. Latest consensus state: current_term: 1 leader_uuid: "66bae122a7154fa58a97062b3d5db0a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } } }
I20260927 11:20:54.207834 15791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:20:54.207837 15792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: This master's current role is: LEADER
W20260927 11:20:54.209280 15806 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:20:54.209342 15806 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:20:54.209421 15796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:20:54.209576 15796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:20:54.211119 15796 catalog_manager.cc:1347] Generated new cluster ID: 2dc46c4f6e1e4008965154ff80de1dd8
I20260927 11:20:54.211172 15796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:20:54 dist-test-slave-lsvg krb5kdc[15704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:54.216996 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.206387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.0.98.62:41273)
0927 11:20:54.206984 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:54.206995 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:54.207031 (+ 36us) server_negotiation.cc:407] Connection header received
0927 11:20:54.207148 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:54.207154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:54.207210 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:54.208205 (+ 995us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:54.208321 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.209852 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.210766 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.211305 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.211424 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.213807 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:54.213811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:54.213812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:54.213823 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:54.215489 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:54.215864 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:54.215867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:54.215868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:54.215975 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:54.216185 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:54.216186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:54.216187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:54.216423 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:54.216582 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:54.216659 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 11:20:54.216810 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":484,"threads_started":1}
I20260927 11:20:54.218160 15796 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:20:54.218840 15796 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:20:54.225390 15796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Generated new TSK 0
I20260927 11:20:54.225553 15796 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:20:54.260303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41273
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:54.357003 15811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:54.357231 15811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:54.357255 15811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:54.358439 15811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:54.358484 15811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:54.358512 15811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:20:54.359990 15811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:54.360234 15811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:54.360491 15811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:54.366434 15816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:54.366460 15817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:54.366484 15819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:54.366804 15811 server_base.cc:1034] running on GCE node
I20260927 11:20:54.367020 15811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:54.367264 15811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:54.368449 15811 hybrid_clock.cc:648] HybridClock initialized: now 1790508054368449 us; error 43 us; skew 500 ppm
Sep 27 11:20:54 dist-test-slave-lsvg krb5kdc[15704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:54.371193 15811 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:20:54.371604 15811 webserver.cc:460] Webserver started at http://127.0.98.1:37943/ using document root <none> and password file <none>
I20260927 11:20:54.371796 15811 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:54.371837 15811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:54.371937 15811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:20:54.373402 15811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "9f55f75218ef48b58e1b327c7fc9610f"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "37ac57d20610953e3333ac578d6f6a0a"
server_key_iv: "a04f1190dbd854e30a44aea5a9b3aec7"
server_key_version: "encryptionkey@0"
I20260927 11:20:54.373869 15811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9f55f75218ef48b58e1b327c7fc9610f"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "37ac57d20610953e3333ac578d6f6a0a"
server_key_iv: "a04f1190dbd854e30a44aea5a9b3aec7"
server_key_version: "encryptionkey@0"
I20260927 11:20:54.375885 15811 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 11:20:54.376976 15826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:54.377202 15811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:20:54.377303 15811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "9f55f75218ef48b58e1b327c7fc9610f"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "37ac57d20610953e3333ac578d6f6a0a"
server_key_iv: "a04f1190dbd854e30a44aea5a9b3aec7"
server_key_version: "encryptionkey@0"
I20260927 11:20:54.377385 15811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:54.400138 15811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:54.402523 15811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:54.402640 15811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:54.402834 15811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:54.403167 15811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:54.403203 15811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:54.403251 15811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:54.403278 15811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:54.409471 15811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38241
I20260927 11:20:54.409511 15939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38241 every 8 connection(s)
I20260927 11:20:54.409847 15811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:20:54 dist-test-slave-lsvg krb5kdc[15704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:54.415894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 15811
I20260927 11:20:54.416034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:20:54.416230 392 external_mini_cluster.cc:1428] Setting key 1d867df82c3abf141919867da7454020
I20260927 11:20:54.419580 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.410521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40441 (local address 127.0.98.62:41273)
0927 11:20:54.410563 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:54.410565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:54.410762 (+ 197us) server_negotiation.cc:407] Connection header received
0927 11:20:54.411242 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:54.411243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:54.411258 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:54.411309 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:54.412211 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.412705 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.413783 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.413995 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.414050 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.416309 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:54.416311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:54.416311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:54.416323 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:54.418152 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:54.418578 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:54.418579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:54.418580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:54.418642 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:54.418913 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:54.418914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:54.418915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:54.419038 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:54.419113 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:54.419393 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 11:20:54.419492 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:20:54.419785 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.410503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:40441)
0927 11:20:54.410643 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:54.410679 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:54.411151 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:54.411363 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:54.411366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:54.411933 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:54.412130 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.412135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.412812 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.412813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.413699 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.413701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.414079 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.414080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.414183 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.414774 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:54.414783 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:54.416206 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:54.418236 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:54.418242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:54.418243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:54.418462 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:54.418718 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:54.418719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:54.418719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:54.418837 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:54.419105 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:54.419112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:54.419259 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 11:20:54.419371 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 11:20:54.419535 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260927 11:20:54.420235 15940 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41273
I20260927 11:20:54.420356 15940 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:54.420534 15940 heartbeater.cc:511] Master 127.0.98.62:41273 requested a full tablet report, sending...
I20260927 11:20:54.421087 15741 ts_manager.cc:194] Registered new tserver with Master: 9f55f75218ef48b58e1b327c7fc9610f (127.0.98.1:38241)
I20260927 11:20:54.421681 15741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40441
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:20:54.443681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41273
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:54.525808 15945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:54.526052 15945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:54.526079 15945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:54.527282 15945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:54.527323 15945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:54.527350 15945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:20:54.528884 15945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:54.529152 15945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:54.529408 15945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:54.534585 15951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:54.534580 15953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:54.534564 15950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:54.534824 15945 server_base.cc:1034] running on GCE node
I20260927 11:20:54.535030 15945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:54.535279 15945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:54.536439 15945 hybrid_clock.cc:648] HybridClock initialized: now 1790508054536432 us; error 49 us; skew 500 ppm
Sep 27 11:20:54 dist-test-slave-lsvg krb5kdc[15704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:54.539122 15945 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:20:54.539662 15945 webserver.cc:460] Webserver started at http://127.0.98.2:32853/ using document root <none> and password file <none>
I20260927 11:20:54.539929 15945 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:54.539999 15945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:54.540163 15945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:20:54.541355 15945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "6d7dac5469b545d2bb2dbdaaf68293ca"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "f75223cf323f07c1f3003899cd62c683"
server_key_iv: "6e9fefe59e6ec099f94b7f22e60e980d"
server_key_version: "encryptionkey@0"
I20260927 11:20:54.541757 15945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6d7dac5469b545d2bb2dbdaaf68293ca"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "f75223cf323f07c1f3003899cd62c683"
server_key_iv: "6e9fefe59e6ec099f94b7f22e60e980d"
server_key_version: "encryptionkey@0"
I20260927 11:20:54.543081 15945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 11:20:54.543810 15960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:54.544001 15945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:20:54.544103 15945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "6d7dac5469b545d2bb2dbdaaf68293ca"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "f75223cf323f07c1f3003899cd62c683"
server_key_iv: "6e9fefe59e6ec099f94b7f22e60e980d"
server_key_version: "encryptionkey@0"
I20260927 11:20:54.544186 15945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:54.560685 15945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:54.562829 15945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:54.563009 15945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:54.563238 15945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:54.563573 15945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:54.563627 15945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:54.563660 15945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:54.563689 15945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:54.569828 15945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42891
I20260927 11:20:54.569867 16073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42891 every 8 connection(s)
I20260927 11:20:54.570197 15945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:20:54 dist-test-slave-lsvg krb5kdc[15704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:54.578759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 15945
I20260927 11:20:54.578872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:20:54.579095 392 external_mini_cluster.cc:1428] Setting key dd7809e518152debd92a12b3e748eca9
I20260927 11:20:54.579288 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.570923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33661 (local address 127.0.98.62:41273)
0927 11:20:54.570994 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:54.570996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:54.571217 (+ 221us) server_negotiation.cc:407] Connection header received
0927 11:20:54.571699 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:54.571700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:54.571709 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:54.571769 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:54.572692 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.573247 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.574208 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.574394 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.574448 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.576595 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:54.576596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:54.576596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:54.576602 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:54.577912 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:54.578281 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:54.578282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:54.578282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:54.578335 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:54.578556 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:54.578556 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:54.578556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:54.578659 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:54.578708 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:54.579076 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 11:20:54.579166 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 11:20:54.579468 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.570905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:33661)
0927 11:20:54.571085 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:54.571128 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:54.571616 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:54.571828 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:54.571831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:54.572415 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:54.572614 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.572618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.573327 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.573327 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.574131 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.574133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.574466 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.574467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.574570 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.575121 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:54.575135 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:54.576516 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:54.577987 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:54.577993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:54.577995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:54.578210 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:54.578394 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:54.578395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:54.578395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:54.578502 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:54.578726 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:54.578733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:54.578909 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 11:20:54.579040 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 11:20:54.579213 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260927 11:20:54.579958 16074 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41273
I20260927 11:20:54.580169 16074 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:54.580353 16074 heartbeater.cc:511] Master 127.0.98.62:41273 requested a full tablet report, sending...
I20260927 11:20:54.580778 15741 ts_manager.cc:194] Registered new tserver with Master: 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891)
I20260927 11:20:54.581313 15741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33661
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:20:54.603660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41273
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:20:54.686367 16079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:20:54.686576 16079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:20:54.686600 16079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:20:54.687788 16079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:20:54.687830 16079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:20:54.687857 16079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:20:54.689352 16079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:20:54.689595 16079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:20:54.689847 16079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:20:54.695267 16087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:54.695205 16084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:20:54.695207 16085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:20:54.695242 16079 server_base.cc:1034] running on GCE node
I20260927 11:20:54.695675 16079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:20:54.695881 16079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:20:54.697021 16079 hybrid_clock.cc:648] HybridClock initialized: now 1790508054697010 us; error 30 us; skew 500 ppm
Sep 27 11:20:54 dist-test-slave-lsvg krb5kdc[15704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:20:54.699497 16079 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:20:54.699977 16079 webserver.cc:460] Webserver started at http://127.0.98.3:35329/ using document root <none> and password file <none>
I20260927 11:20:54.700201 16079 fs_manager.cc:359] Metadata directory not provided
I20260927 11:20:54.700266 16079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:20:54.700420 16079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:20:54.701539 16079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "61c5aa0fe737498982d260dd4bc164d6"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "be5b5e8442896e978afa2b524b027c65"
server_key_iv: "76ae0541309bea92ea1c55afaa6cc454"
server_key_version: "encryptionkey@0"
I20260927 11:20:54.701901 16079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "61c5aa0fe737498982d260dd4bc164d6"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "be5b5e8442896e978afa2b524b027c65"
server_key_iv: "76ae0541309bea92ea1c55afaa6cc454"
server_key_version: "encryptionkey@0"
I20260927 11:20:54.703244 16079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:20:54.703986 16094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:54.704149 16079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:20:54.704253 16079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "61c5aa0fe737498982d260dd4bc164d6"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "be5b5e8442896e978afa2b524b027c65"
server_key_iv: "76ae0541309bea92ea1c55afaa6cc454"
server_key_version: "encryptionkey@0"
I20260927 11:20:54.704325 16079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:20:54.732788 16079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:20:54.734774 16079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:20:54.734939 16079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:20:54.735184 16079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:20:54.735533 16079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:20:54.735589 16079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:54.735632 16079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:20:54.735657 16079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:20:54.741752 16079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46129
I20260927 11:20:54.741843 16207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46129 every 8 connection(s)
I20260927 11:20:54.742133 16079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 11:20:54 dist-test-slave-lsvg krb5kdc[15704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:20:54.748905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 16079
I20260927 11:20:54.748993 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:20:54.749191 392 external_mini_cluster.cc:1428] Setting key 947174ae68a344bda0d001786128564f
I20260927 11:20:54.751394 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.742859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53531 (local address 127.0.98.62:41273)
0927 11:20:54.742905 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:54.742907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:54.743142 (+ 235us) server_negotiation.cc:407] Connection header received
0927 11:20:54.743672 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:54.743673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:54.743687 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:54.743745 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:54.744716 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.745282 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.746318 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.746519 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.746605 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.748819 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:54.748820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:54.748821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:54.748827 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:54.750290 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:54.750668 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:54.750669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:54.750669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:54.750722 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:54.750889 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:54.750889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:54.750890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:54.750987 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:54.751036 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:54.751269 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 11:20:54.751329 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:20:54.751562 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.742850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:53531)
0927 11:20:54.743019 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:54.743055 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:54.743566 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:54.743816 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:54.743820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:54.744413 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:20:54.744611 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.744616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.745405 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.745406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.746224 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.746226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.746614 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.746615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.746746 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.747264 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:20:54.747272 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:20:54.748698 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:20:54.750383 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:54.750388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:54.750390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:54.750606 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:54.750762 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:20:54.750763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:20:54.750763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:20:54.750844 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:20:54.751043 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:20:54.751047 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:20:54.751160 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 11:20:54.751241 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 11:20:54.751358 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260927 11:20:54.752074 16208 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41273
I20260927 11:20:54.752170 16208 heartbeater.cc:464] Registering TS with master...
I20260927 11:20:54.752357 16208 heartbeater.cc:511] Master 127.0.98.62:41273 requested a full tablet report, sending...
I20260927 11:20:54.752714 15741 ts_manager.cc:194] Registered new tserver with Master: 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129)
I20260927 11:20:54.753139 15741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53531
I20260927 11:20:54.761195 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:20:54.768441 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.762589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.0.98.62:41273)
0927 11:20:54.762667 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:54.762669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:54.762781 (+ 112us) server_negotiation.cc:407] Connection header received
0927 11:20:54.762883 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:54.762884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:54.762895 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:54.762951 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:20:54.763510 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.763992 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.764709 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.764919 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.765011 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.766135 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:20:54.766136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:20:54.766137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:20:54.766144 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:20:54.767289 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:54.767628 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:54.767629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:54.767629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:54.767687 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:20:54.767885 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:20:54.767886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:20:54.767887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:20:54.768034 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:20:54.768139 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:54.768259 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 11:20:54.768338 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:20:54.769934 15741 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:20:54.771422 15741 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:20:54.777798 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.773668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46129 (local address 127.0.0.1:59864)
0927 11:20:54.773806 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:54.773882 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:54.773952 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:54.774241 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:54.774244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:54.774247 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:54.774453 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.774456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.775432 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.775433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.776646 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.776649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.777481 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.777482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.777573 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.777588 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 11:20:54.777666 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 11:20:54.777717 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260927 11:20:54.777834 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.773782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.0.98.3:46129)
0927 11:20:54.773997 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:54.774002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:54.774012 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:20:54.774053 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:54.774055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:54.774071 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:54.774147 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:54.774542 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.775312 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.776725 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.777394 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.777472 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.777533 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:54.777693 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 11:20:54.777735 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":99,"threads_started":1}
I20260927 11:20:54.778191 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.774072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38241 (local address 127.0.0.1:40932)
0927 11:20:54.774268 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:54.774276 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:54.774318 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:54.774628 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:54.774629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:54.774630 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:54.774671 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.774672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.775925 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.775926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.776916 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.776918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.777937 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.777938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.778007 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.778012 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:20:54.778066 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 11:20:54.778107 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":55,"threads_started":1}
I20260927 11:20:54.778226 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.774194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.0.98.1:38241)
0927 11:20:54.774395 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:54.774400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:54.774410 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:20:54.774454 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:54.774456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:54.774471 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:54.774548 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:54.774756 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.775827 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.777009 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.777771 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.777841 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.777897 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:54.778102 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 11:20:54.778147 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20260927 11:20:54.779027 16142 tablet_service.cc:1467] Processing CreateTablet for tablet 445ac63532eb41948e29166519f52444 (DEFAULT_TABLE table=test-table [id=ca94f69de43541ea8da68a5f03f4ac67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:20:54.779023 15874 tablet_service.cc:1467] Processing CreateTablet for tablet 445ac63532eb41948e29166519f52444 (DEFAULT_TABLE table=test-table [id=ca94f69de43541ea8da68a5f03f4ac67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:20:54.779407 15874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445ac63532eb41948e29166519f52444. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:54.779405 16142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445ac63532eb41948e29166519f52444. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:54.779458 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.773893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42891 (local address 127.0.0.1:51188)
0927 11:20:54.774178 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:54.774188 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:54.774235 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:54.774533 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:54.774534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:54.774536 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:54.774583 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.774585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.776996 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.776996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.777947 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.777950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.779205 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.779206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.779269 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.779273 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:20:54.779325 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 11:20:54.779379 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":86,"threads_started":1}
I20260927 11:20:54.779481 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.774014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51188 (local address 127.0.98.2:42891)
0927 11:20:54.774206 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:54.774211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:54.774224 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:20:54.774336 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:54.774339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:54.774356 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:54.774435 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:54.774672 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.775815 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.778089 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.779037 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.779089 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.779140 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:54.779346 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 11:20:54.779393 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
I20260927 11:20:54.782889 16226 tablet_bootstrap.cc:492] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Bootstrap starting.
I20260927 11:20:54.782999 16225 tablet_bootstrap.cc:492] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6: Bootstrap starting.
I20260927 11:20:54.783663 16225 tablet_bootstrap.cc:654] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:54.783676 16226 tablet_bootstrap.cc:654] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:54.783667 16008 tablet_service.cc:1467] Processing CreateTablet for tablet 445ac63532eb41948e29166519f52444 (DEFAULT_TABLE table=test-table [id=ca94f69de43541ea8da68a5f03f4ac67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:20:54.784109 16008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445ac63532eb41948e29166519f52444. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:20:54.784260 16225 log.cc:824] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:54.784287 16226 log.cc:824] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:54.785002 16225 tablet_bootstrap.cc:492] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6: No bootstrap required, opened a new log
I20260927 11:20:54.785100 16226 tablet_bootstrap.cc:492] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: No bootstrap required, opened a new log
I20260927 11:20:54.785120 16225 ts_tablet_manager.cc:1397] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:20:54.785174 16226 ts_tablet_manager.cc:1397] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:20:54.786443 16225 raft_consensus.cc:348] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:20:54.786604 16225 raft_consensus.cc:374] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:54.786648 16225 raft_consensus.cc:729] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c5aa0fe737498982d260dd4bc164d6, State: Initialized, Role: FOLLOWER
I20260927 11:20:54.786798 16225 consensus_queue.cc:260] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:20:54.787107 16208 heartbeater.cc:503] Master 127.0.98.62:41273 was elected leader, sending a full tablet report...
I20260927 11:20:54.787102 16225 ts_tablet_manager.cc:1428] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:20:54.787401 16226 raft_consensus.cc:348] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:20:54.787585 16226 raft_consensus.cc:374] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:54.787629 16226 raft_consensus.cc:729] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55f75218ef48b58e1b327c7fc9610f, State: Initialized, Role: FOLLOWER
I20260927 11:20:54.787758 16226 consensus_queue.cc:260] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:20:54.787889 16230 tablet_bootstrap.cc:492] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: Bootstrap starting.
I20260927 11:20:54.788055 16226 ts_tablet_manager.cc:1428] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:20:54.788075 15940 heartbeater.cc:503] Master 127.0.98.62:41273 was elected leader, sending a full tablet report...
I20260927 11:20:54.788669 16230 tablet_bootstrap.cc:654] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: Neither blocks nor log segments found. Creating new log.
I20260927 11:20:54.789292 16230 log.cc:824] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: Log is configured to *not* fsync() on all Append() calls
I20260927 11:20:54.789920 16230 tablet_bootstrap.cc:492] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: No bootstrap required, opened a new log
I20260927 11:20:54.789973 16230 ts_tablet_manager.cc:1397] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:20:54.791211 16230 raft_consensus.cc:348] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:20:54.791342 16230 raft_consensus.cc:374] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:20:54.791366 16230 raft_consensus.cc:729] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7dac5469b545d2bb2dbdaaf68293ca, State: Initialized, Role: FOLLOWER
I20260927 11:20:54.791469 16230 consensus_queue.cc:260] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:20:54.791738 16230 ts_tablet_manager.cc:1428] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:20:54.791759 16074 heartbeater.cc:503] Master 127.0.98.62:41273 was elected leader, sending a full tablet report...
W20260927 11:20:54.820755 16075 tablet.cc:2369] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:20:54.895010 16233 raft_consensus.cc:482] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:20:54.895129 16233 raft_consensus.cc:504] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:20:54.895494 16233 leader_election.cc:288] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129), 9f55f75218ef48b58e1b327c7fc9610f (127.0.98.1:38241)
I20260927 11:20:54.898768 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.895872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56663 (local address 127.0.98.1:38241)
0927 11:20:54.895937 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:54.895939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:54.896091 (+ 152us) server_negotiation.cc:407] Connection header received
0927 11:20:54.896153 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:54.896154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:54.896168 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:54.896221 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:54.896357 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.897029 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.898041 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.898425 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.898507 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.898517 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:54.898661 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 11:20:54.898698 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 11:20:54.898765 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.895765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33197 (local address 127.0.98.3:46129)
0927 11:20:54.895862 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:54.895864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:54.895874 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:20:54.895905 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:54.895906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:54.895929 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:54.895985 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:20:54.896155 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.896901 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.898002 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.898394 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.898483 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.898494 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:54.898653 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 11:20:54.898692 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 11:20:54.898774 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.895658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46129 (local address 127.0.98.2:33197)
0927 11:20:54.895714 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:54.895727 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:54.895772 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:54.895999 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:54.896000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:54.896003 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:54.896084 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.896085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.897005 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.897006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.897905 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.897908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.898480 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.898481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.898560 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.898568 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 11:20:54.898645 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 11:20:54.898689 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":18}
I20260927 11:20:54.898788 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.895811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38241 (local address 127.0.98.2:56663)
0927 11:20:54.896001 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:54.896011 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:20:54.896085 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:20:54.896244 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:20:54.896244 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:20:54.896245 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:20:54.896292 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.896293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.897124 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.897125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.897948 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:20:54.897950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.898508 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.898509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:20:54.898587 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.898591 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:20:54.898656 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 11:20:54.898698 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":130,"threads_started":1}
I20260927 11:20:54.899161 16162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445ac63532eb41948e29166519f52444" candidate_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c5aa0fe737498982d260dd4bc164d6" is_pre_election: true
I20260927 11:20:54.899158 15894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445ac63532eb41948e29166519f52444" candidate_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55f75218ef48b58e1b327c7fc9610f" is_pre_election: true
I20260927 11:20:54.899322 16162 raft_consensus.cc:2454] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d7dac5469b545d2bb2dbdaaf68293ca in term 0.
I20260927 11:20:54.899322 15894 raft_consensus.cc:2454] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d7dac5469b545d2bb2dbdaaf68293ca in term 0.
I20260927 11:20:54.899567 15962 leader_election.cc:302] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61c5aa0fe737498982d260dd4bc164d6, 6d7dac5469b545d2bb2dbdaaf68293ca; no voters:
I20260927 11:20:54.899690 16233 raft_consensus.cc:2781] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:20:54.899783 16233 raft_consensus.cc:482] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:20:54.899817 16233 raft_consensus.cc:3037] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:54.900543 16233 raft_consensus.cc:504] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:20:54.900682 16233 leader_election.cc:288] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [CANDIDATE]: Term 1 election: Requested vote from peers 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129), 9f55f75218ef48b58e1b327c7fc9610f (127.0.98.1:38241)
I20260927 11:20:54.900866 16162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445ac63532eb41948e29166519f52444" candidate_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c5aa0fe737498982d260dd4bc164d6"
I20260927 11:20:54.900923 15894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445ac63532eb41948e29166519f52444" candidate_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55f75218ef48b58e1b327c7fc9610f"
I20260927 11:20:54.900969 16162 raft_consensus.cc:3037] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:54.901024 15894 raft_consensus.cc:3037] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:20:54.901738 16162 raft_consensus.cc:2454] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d7dac5469b545d2bb2dbdaaf68293ca in term 1.
I20260927 11:20:54.901753 15894 raft_consensus.cc:2454] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d7dac5469b545d2bb2dbdaaf68293ca in term 1.
I20260927 11:20:54.901901 15962 leader_election.cc:302] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d7dac5469b545d2bb2dbdaaf68293ca, 9f55f75218ef48b58e1b327c7fc9610f; no voters:
I20260927 11:20:54.902031 16233 raft_consensus.cc:2781] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:20:54.902168 16233 raft_consensus.cc:686] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 1 LEADER]: Becoming Leader. State: Replica: 6d7dac5469b545d2bb2dbdaaf68293ca, State: Running, Role: LEADER
I20260927 11:20:54.902381 16233 consensus_queue.cc:237] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:20:54.903272 15749 catalog_manager.cc:5495] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } health_report { overall_health: UNKNOWN } } }
W20260927 11:20:54.910549 15941 tablet.cc:2369] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:20:54.925357 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.923121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.0.98.2:42891)
0927 11:20:54.923184 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:20:54.923186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:20:54.923195 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:20:54.923239 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:20:54.923240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:20:54.923260 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:20:54.923326 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:20:54.923463 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.923913 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.924640 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:20:54.924818 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:20:54.924866 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:20:54.925002 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:20:54.925165 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:20:54.925230 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:20:54.925280 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 11:20:54.925299 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 11:20:54.927152 15894 raft_consensus.cc:1261] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Refusing update from remote peer 6d7dac5469b545d2bb2dbdaaf68293ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:20:54.927204 16162 raft_consensus.cc:1261] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6 [term 1 FOLLOWER]: Refusing update from remote peer 6d7dac5469b545d2bb2dbdaaf68293ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:20:54.927536 16235 consensus_queue.cc:1035] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:20:54.927665 16233 consensus_queue.cc:1035] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:20:54.930333 16240 mvcc.cc:205] Tried to move back new op lower bound from 7333920992978862080 to 7333920992880766976. Current Snapshot: MvccSnapshot[applied={T|T < 7333920992978862080}]
I20260927 11:20:54.930490 16239 mvcc.cc:205] Tried to move back new op lower bound from 7333920992978862080 to 7333920992880766976. Current Snapshot: MvccSnapshot[applied={T|T < 7333920992978862080}]
I20260927 11:20:54.932293 15741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:20:54.932401 15741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:20:54.933176 15741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Sending DeleteTablet for 3 replicas of tablet 445ac63532eb41948e29166519f52444
I20260927 11:20:54.933542 16142 tablet_service.cc:1514] Processing DeleteTablet for tablet 445ac63532eb41948e29166519f52444 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59864
I20260927 11:20:54.933573 16008 tablet_service.cc:1514] Processing DeleteTablet for tablet 445ac63532eb41948e29166519f52444 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51188
I20260927 11:20:54.933738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 15811
I20260927 11:20:54.933760 15874 tablet_service.cc:1514] Processing DeleteTablet for tablet 445ac63532eb41948e29166519f52444 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40932
I20260927 11:20:54.937192 16248 tablet_replica.cc:330] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6: stopping tablet replica
I20260927 11:20:54.937311 16248 raft_consensus.cc:2229] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:20:54.937426 16248 raft_consensus.cc:2258] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:20:54.937599 16249 tablet_replica.cc:330] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: stopping tablet replica
I20260927 11:20:54.937700 16249 raft_consensus.cc:2229] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:20:54.937808 16249 raft_consensus.cc:2258] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:20:54.938089 16248 ts_tablet_manager.cc:1905] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:20:54.938241 16249 ts_tablet_manager.cc:1905] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 11:20:54.939469 15734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38241 (error 108)
I20260927 11:20:54.940164 16248 ts_tablet_manager.cc:1918] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:20:54.940243 16248 log.cc:1196] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/445ac63532eb41948e29166519f52444
I20260927 11:20:54.940328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 15945
I20260927 11:20:54.940521 16248 ts_tablet_manager.cc:1939] T 445ac63532eb41948e29166519f52444 P 61c5aa0fe737498982d260dd4bc164d6: Deleting consensus metadata
I20260927 11:20:54.940980 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.940207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38241 (local address 127.0.0.1:40944)
0927 11:20:54.940882 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:54.940944 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":647}
W20260927 11:20:54.943053 15734 catalog_manager.cc:4584] TS 9f55f75218ef48b58e1b327c7fc9610f (127.0.98.1:38241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 445ac63532eb41948e29166519f52444: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38241: connect: Connection refused (error 111)
I20260927 11:20:54.943150 15734 catalog_manager.cc:4841] TS 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129): tablet 445ac63532eb41948e29166519f52444 (table test-table [id=ca94f69de43541ea8da68a5f03f4ac67]) successfully deleted
I20260927 11:20:54.946710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 16079
I20260927 11:20:54.946728 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:20:54.946596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42891 (local address 127.0.0.1:51208)
0927 11:20:54.946658 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 11:20:54.946691 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 11:20:54.951177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 15717
I20260927 11:21:14.956470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41273
--webserver_interface=127.0.98.62
--webserver_port=43855
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:15.040278 16251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:15.040486 16251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:15.040508 16251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:15.041695 16251 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 11:21:15.041742 16251 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 11:21:15.041755 16251 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 11:21:15.041767 16251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:21:15.041780 16251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:15.041792 16251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:15.041847 16251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:21:15.041865 16251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:21:15.043357 16251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:15.043627 16251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:15.043857 16251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:15.052902 16260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:15.052902 16257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:15.053141 16258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:15.052984 16251 server_base.cc:1034] running on GCE node
I20260927 11:21:15.053426 16251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:15.053736 16251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:15.054870 16251 hybrid_clock.cc:648] HybridClock initialized: now 1790508075054858 us; error 26 us; skew 500 ppm
Sep 27 11:21:15 dist-test-slave-lsvg krb5kdc[15704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:15.062973 16251 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:21:15.063457 16251 webserver.cc:460] Webserver started at http://127.0.98.62:43855/ using document root <none> and password file <none>
I20260927 11:21:15.063697 16251 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:15.063776 16251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:15.065229 16251 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:15.065915 16267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:15.066072 16251 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:21:15.066169 16251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "66bae122a7154fa58a97062b3d5db0a9"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "b04c2a4f3d8dd4f8f01d8f5bfb0e4021"
server_key_iv: "08508527e82c5380d9e997368799e843"
server_key_version: "encryptionkey@0"
I20260927 11:21:15.066572 16251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:15.083087 16251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:15.085316 16251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:15.085479 16251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:15.089572 16251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41273
I20260927 11:21:15.089609 16319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41273 every 8 connection(s)
I20260927 11:21:15.090027 16251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:21:15.091475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 16251
I20260927 11:21:15.091629 16320 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:21:15.091817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38241
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41273
--webserver_port=37943
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:21:15.092672 16320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Bootstrap starting.
I20260927 11:21:15.101413 16320 log.cc:824] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:15.103370 16320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:21:15.103685 16320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Bootstrap complete.
I20260927 11:21:15.106180 16320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } }
I20260927 11:21:15.106438 16320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bae122a7154fa58a97062b3d5db0a9, State: Initialized, Role: FOLLOWER
I20260927 11:21:15.106608 16320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } }
I20260927 11:21:15.106678 16320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:21:15.106743 16320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:21:15.106798 16320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:21:15.107918 16320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } }
I20260927 11:21:15.108042 16320 leader_election.cc:302] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66bae122a7154fa58a97062b3d5db0a9; no voters:
I20260927 11:21:15.108270 16320 leader_election.cc:288] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 11:21:15.108310 16324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 11:21:15.108436 16324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 2 LEADER]: Becoming Leader. State: Replica: 66bae122a7154fa58a97062b3d5db0a9, State: Running, Role: LEADER
I20260927 11:21:15.108525 16324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } }
I20260927 11:21:15.108558 16320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:21:15.108990 16325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "66bae122a7154fa58a97062b3d5db0a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } } }
I20260927 11:21:15.109122 16325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:21:15.109521 16331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:21:15.110236 16326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66bae122a7154fa58a97062b3d5db0a9. Latest consensus state: current_term: 2 leader_uuid: "66bae122a7154fa58a97062b3d5db0a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } } }
I20260927 11:21:15.110357 16326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:21:15.110431 16331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:21:15.110543 16331 catalog_manager.cc:1259] Loaded cluster ID: 2dc46c4f6e1e4008965154ff80de1dd8
I20260927 11:21:15.110586 16331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:21:15.111632 16331 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:21:15.111829 16331 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Loaded TSK: 0
I20260927 11:21:15.123543 16331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Generated new TSK 1
I20260927 11:21:15.123669 16331 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 11:21:15.189994 16322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:15.190215 16322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:15.190240 16322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:15.191418 16322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:15.191459 16322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:15.191486 16322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:21:15.192924 16322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37943
--tserver_master_addrs=127.0.98.62:41273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:15.193188 16322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:15.193449 16322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:15.198567 16350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:15.198700 16347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:15.198582 16348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:15.199378 16322 server_base.cc:1034] running on GCE node
I20260927 11:21:15.199601 16322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:15.199900 16322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:15.201097 16322 hybrid_clock.cc:648] HybridClock initialized: now 1790508075201079 us; error 36 us; skew 500 ppm
Sep 27 11:21:15 dist-test-slave-lsvg krb5kdc[15704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:15.204041 16322 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:21:15.204515 16322 webserver.cc:460] Webserver started at http://127.0.98.1:37943/ using document root <none> and password file <none>
I20260927 11:21:15.204754 16322 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:15.204836 16322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:15.206403 16322 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:15.207129 16357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:15.207315 16322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:15.207410 16322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "9f55f75218ef48b58e1b327c7fc9610f"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "37ac57d20610953e3333ac578d6f6a0a"
server_key_iv: "a04f1190dbd854e30a44aea5a9b3aec7"
server_key_version: "encryptionkey@0"
I20260927 11:21:15.207778 16322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:15.222150 16322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:15.224433 16322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:15.224589 16322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:15.224819 16322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:15.225358 16364 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 11:21:15.226758 16322 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 11:21:15.226809 16322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 11:21:15.226843 16322 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 11:21:15.227490 16322 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 11:21:15.227533 16322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 11:21:15.227578 16364 tablet_bootstrap.cc:492] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Bootstrap starting.
I20260927 11:21:15.235378 16322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38241
I20260927 11:21:15.235430 16471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38241 every 8 connection(s)
I20260927 11:21:15.235906 16322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:21:15.237206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 16322
I20260927 11:21:15.237560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42891
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41273
--webserver_port=32853
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:21:15.239142 16364 log.cc:824] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:15.241221 16364 tablet_bootstrap.cc:492] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:21:15.241525 16364 tablet_bootstrap.cc:492] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Bootstrap complete.
I20260927 11:21:15.242725 16364 ts_tablet_manager.cc:1397] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20260927 11:21:15.244201 16364 raft_consensus.cc:348] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:15.244551 16364 raft_consensus.cc:729] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55f75218ef48b58e1b327c7fc9610f, State: Initialized, Role: FOLLOWER
I20260927 11:21:15.244706 16364 consensus_queue.cc:260] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:15.244969 16364 ts_tablet_manager.cc:1428] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Sep 27 11:21:15 dist-test-slave-lsvg krb5kdc[15704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:21:15.249512 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.236650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:51407)
0927 11:21:15.236930 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:15.236973 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:15.237943 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:15.238686 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:15.238690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:15.239054 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:15.239324 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.239330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.241141 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.241142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.242679 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.242682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.243511 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.243513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.243641 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.244221 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:15.244231 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:15.245995 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:15.247924 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:15.247931 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:15.247934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:15.248151 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:15.248394 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:15.248395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:15.248395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:15.248504 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:15.248872 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:15.248878 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:15.249009 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 11:21:15.249151 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 11:21:15.249331 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":93,"threads_started":1}
I20260927 11:21:15.249536 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.236764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51407 (local address 127.0.98.62:41273)
0927 11:21:15.238265 (+ 1501us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:15.238273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:15.238308 (+ 35us) server_negotiation.cc:407] Connection header received
0927 11:21:15.238437 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:15.238444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:15.238497 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:15.239463 (+ 966us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:15.239545 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.240991 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.242785 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.243395 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.243520 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.246115 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:15.246119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:15.246119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:15.246131 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:15.247823 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:15.248232 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:15.248236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:15.248236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:15.248329 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:15.248575 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:15.248576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:15.248576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:15.248791 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:15.248883 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:15.249203 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 11:21:15.249321 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1450,"thread_start_us":81,"threads_started":1}
I20260927 11:21:15.250365 16472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41273
I20260927 11:21:15.250460 16472 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:15.250624 16472 heartbeater.cc:511] Master 127.0.98.62:41273 requested a full tablet report, sending...
I20260927 11:21:15.251297 16284 ts_manager.cc:194] Registered new tserver with Master: 9f55f75218ef48b58e1b327c7fc9610f (127.0.98.1:38241)
I20260927 11:21:15.252434 16284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51407
I20260927 11:21:15.253388 16472 heartbeater.cc:503] Master 127.0.98.62:41273 was elected leader, sending a full tablet report...
Sep 27 11:21:15 dist-test-slave-lsvg krb5kdc[15704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:21:15.260059 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.252118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.0.98.1:38241)
0927 11:21:15.252449 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:15.252453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:15.252462 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:21:15.252495 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:15.252567 (+ 72us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:15.252584 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:15.252646 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:15.253297 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.254168 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.254920 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.255249 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.255340 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.257324 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:15.257331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:15.257331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:15.257338 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:15.258804 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:15.259201 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:15.259204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:15.259204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:15.259258 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:15.259457 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:15.259457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:15.259458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:15.259607 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:15.259697 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:15.259883 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 11:21:15.259972 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":57,"threads_started":1}
I20260927 11:21:15.260059 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.251890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38241 (local address 127.0.0.1:45466)
0927 11:21:15.252245 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:15.252278 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:15.252335 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:15.252766 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:15.252769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:15.253128 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:15.253209 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.253211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.254246 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.254247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.254824 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.254826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.255351 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.255354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.255434 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.255903 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:15.255907 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:15.257216 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:15.258901 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:15.258908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:15.258909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:15.259103 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:15.259329 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:15.259330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:15.259331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:15.259391 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:15.259677 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:15.259683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:15.259736 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 11:21:15.259845 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 11:21:15.259959 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":90,"threads_started":1}
I20260927 11:21:15.260691 16406 tablet_service.cc:1514] Processing DeleteTablet for tablet 445ac63532eb41948e29166519f52444 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:20:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45466
I20260927 11:21:15.261219 16481 tablet_replica.cc:330] stopping tablet replica
I20260927 11:21:15.261332 16481 raft_consensus.cc:2229] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:21:15.261405 16481 raft_consensus.cc:2258] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:15.261698 16481 ts_tablet_manager.cc:1905] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:15.263962 16481 ts_tablet_manager.cc:1918] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:15.264045 16481 log.cc:1196] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal/wals/445ac63532eb41948e29166519f52444
I20260927 11:21:15.264254 16481 ts_tablet_manager.cc:1939] T 445ac63532eb41948e29166519f52444 P 9f55f75218ef48b58e1b327c7fc9610f: Deleting consensus metadata
I20260927 11:21:15.264716 16269 catalog_manager.cc:4841] TS 9f55f75218ef48b58e1b327c7fc9610f (127.0.98.1:38241): tablet 445ac63532eb41948e29166519f52444 (table test-table [id=ca94f69de43541ea8da68a5f03f4ac67]) successfully deleted
W20260927 11:21:15.333154 16476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:15.333370 16476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:15.333395 16476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:15.334563 16476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:15.334609 16476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:15.334636 16476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:21:15.336074 16476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=32853
--tserver_master_addrs=127.0.98.62:41273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:15.336354 16476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:15.336597 16476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:15.341552 16489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:15.341586 16486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:15.341555 16487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:15.341570 16476 server_base.cc:1034] running on GCE node
I20260927 11:21:15.341892 16476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:15.342133 16476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:15.343284 16476 hybrid_clock.cc:648] HybridClock initialized: now 1790508075343275 us; error 35 us; skew 500 ppm
Sep 27 11:21:15 dist-test-slave-lsvg krb5kdc[15704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:15.345745 16476 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:21:15.346169 16476 webserver.cc:460] Webserver started at http://127.0.98.2:32853/ using document root <none> and password file <none>
I20260927 11:21:15.346395 16476 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:15.346490 16476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:15.347839 16476 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:15.348482 16496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:15.348681 16476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:15.348754 16476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "6d7dac5469b545d2bb2dbdaaf68293ca"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "f75223cf323f07c1f3003899cd62c683"
server_key_iv: "6e9fefe59e6ec099f94b7f22e60e980d"
server_key_version: "encryptionkey@0"
I20260927 11:21:15.349140 16476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:15.368132 16476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:15.370214 16476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:15.370339 16476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:15.370576 16476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:15.370946 16503 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 11:21:15.372393 16503 ts_tablet_manager.cc:1852] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 11:21:15.372684 16503 ts_tablet_manager.cc:1905] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:15.374984 16503 ts_tablet_manager.cc:1918] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:15.375085 16503 log.cc:1196] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/445ac63532eb41948e29166519f52444
I20260927 11:21:15.375381 16503 ts_tablet_manager.cc:1939] T 445ac63532eb41948e29166519f52444 P 6d7dac5469b545d2bb2dbdaaf68293ca: Deleting consensus metadata
I20260927 11:21:15.375600 16476 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 11:21:15.375648 16476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 11:21:15.375705 16476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:15.375747 16476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:15.381582 16476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42891
I20260927 11:21:15.381633 16610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42891 every 8 connection(s)
I20260927 11:21:15.381937 16476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:21:15.387102 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 16476
I20260927 11:21:15.387401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46129
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41273
--webserver_port=35329
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 11:21:15 dist-test-slave-lsvg krb5kdc[15704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:21:15.390969 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.382610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34599 (local address 127.0.98.62:41273)
0927 11:21:15.382683 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:15.382685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:15.382954 (+ 269us) server_negotiation.cc:407] Connection header received
0927 11:21:15.383552 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:15.383553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:15.383563 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:15.383623 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:15.384241 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.384767 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.385743 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.385937 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.385987 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.388115 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:15.388117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:15.388117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:15.388125 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:15.389642 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:15.390009 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:15.390011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:15.390011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:15.390072 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:15.390293 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:15.390294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:15.390294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:15.390441 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:15.390510 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:15.390746 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 11:21:15.390857 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 11:21:15.391064 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.382607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:34599)
0927 11:21:15.382799 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:15.382852 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:15.383368 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:15.383695 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:15.383698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:15.383968 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:15.384160 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.384165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.384846 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.384847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.385673 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.385675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.386005 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.386005 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.386108 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.386582 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:15.386591 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:15.388026 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:15.389717 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:15.389724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:15.389725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:15.389933 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:15.390137 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:15.390137 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:15.390138 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:15.390236 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:15.390505 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:15.390510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:15.390622 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 11:21:15.390702 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 11:21:15.390823 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":115,"threads_started":1}
I20260927 11:21:15.391541 16611 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41273
I20260927 11:21:15.391647 16611 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:15.391849 16611 heartbeater.cc:511] Master 127.0.98.62:41273 requested a full tablet report, sending...
I20260927 11:21:15.392244 16284 ts_manager.cc:194] Registered new tserver with Master: 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891)
I20260927 11:21:15.392784 16284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34599
W20260927 11:21:15.474826 16614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:15.475054 16614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:15.475086 16614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:15.476194 16614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:15.476235 16614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:15.476263 16614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:21:15.477777 16614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35329
--tserver_master_addrs=127.0.98.62:41273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:15.477962 16614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:15.478134 16614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:15.483126 16622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:15.483127 16620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:15.483182 16619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:15.483172 16614 server_base.cc:1034] running on GCE node
I20260927 11:21:15.483484 16614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:15.483690 16614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:15.484822 16614 hybrid_clock.cc:648] HybridClock initialized: now 1790508075484807 us; error 32 us; skew 500 ppm
Sep 27 11:21:15 dist-test-slave-lsvg krb5kdc[15704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:15.487222 16614 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:21:15.487651 16614 webserver.cc:460] Webserver started at http://127.0.98.3:35329/ using document root <none> and password file <none>
I20260927 11:21:15.487867 16614 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:15.487938 16614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:15.489310 16614 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:15.489904 16629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:15.490058 16614 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:15.490154 16614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "61c5aa0fe737498982d260dd4bc164d6"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "be5b5e8442896e978afa2b524b027c65"
server_key_iv: "76ae0541309bea92ea1c55afaa6cc454"
server_key_version: "encryptionkey@0"
I20260927 11:21:15.490505 16614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:15.501881 16614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:15.503762 16614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:15.503932 16614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:15.504166 16614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:15.504482 16614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:15.504531 16614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:15.504571 16614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:15.504596 16614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:15.510548 16614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46129
I20260927 11:21:15.510587 16742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46129 every 8 connection(s)
I20260927 11:21:15.510988 16614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:21:15.512733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 16614
Sep 27 11:21:15 dist-test-slave-lsvg krb5kdc[15704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:21:15.520974 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.513174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.0.98.62:41273)
0927 11:21:15.513291 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:15.513292 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:15.514004 (+ 712us) server_negotiation.cc:407] Connection header received
0927 11:21:15.514072 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:15.514073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:15.514082 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:15.514144 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:15.514845 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.515563 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.516525 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.516826 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.516880 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.518184 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:15.518185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:15.518185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:15.518191 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:15.519823 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:15.520208 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:15.520209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:15.520209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:15.520260 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:15.520450 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:15.520451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:15.520451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:15.520545 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:15.520729 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:15.520766 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 11:21:15.520868 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
I20260927 11:21:15.521914 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.512000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49671 (local address 127.0.98.62:41273)
0927 11:21:15.512050 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:15.512052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:15.512064 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:21:15.512598 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:15.512599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:15.512613 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:15.512668 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:15.513849 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.514660 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.516030 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.516327 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.516421 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.518912 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:15.518913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:15.518913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:15.518921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:15.520489 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:15.520883 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:15.520884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:15.520884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:15.520947 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:15.521263 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:15.521264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:15.521265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:15.521397 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:15.521461 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:15.521747 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 11:21:15.521820 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:21:15.522122 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.511732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:49671)
0927 11:21:15.511924 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:15.511967 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:15.512471 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:15.512742 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:15.512746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:15.513540 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:15.513738 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.513743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.514789 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.514790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.515948 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.515951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.516422 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.516423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.516546 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.517194 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:15.517219 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:15.518804 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:15.520575 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:15.520582 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:15.520585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:15.520802 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:15.521022 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:15.521023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:15.521023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:15.521200 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:15.521456 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:15.521462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:15.521618 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 11:21:15.521718 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 11:21:15.521885 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260927 11:21:15.522544 16743 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41273
I20260927 11:21:15.522644 16743 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:15.522816 16743 heartbeater.cc:511] Master 127.0.98.62:41273 requested a full tablet report, sending...
I20260927 11:21:15.523145 16284 ts_manager.cc:194] Registered new tserver with Master: 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129)
I20260927 11:21:15.523511 16284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49671
I20260927 11:21:15.532615 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:21:15.535960 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.533101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.0.98.62:41273)
0927 11:21:15.533155 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:15.533158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:15.533256 (+ 98us) server_negotiation.cc:407] Connection header received
0927 11:21:15.533347 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:15.533348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:15.533404 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:15.533497 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:21:15.533642 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.534151 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.534958 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.535170 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.535257 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.535440 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:21:15.535496 (+ 56us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 11:21:15.535803 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56248: token expired
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 11:21:15.536039 16475 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56248: token expired
W20260927 11:21:15.536101 16211 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:41273)
I20260927 11:21:15.542704 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.536359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.0.98.62:41273)
0927 11:21:15.536404 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:15.536406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:15.536433 (+ 27us) server_negotiation.cc:407] Connection header received
0927 11:21:15.536498 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:15.536499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:15.536544 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:15.536639 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:15.537256 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.537720 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.538679 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.538873 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.538954 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.540138 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:15.540139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:15.540140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:15.540147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:15.541585 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:15.541938 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:15.541939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:15.541939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:15.541995 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:15.542210 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:15.542211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:15.542211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:15.542322 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:15.542425 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:15.542525 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 11:21:15.542600 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:21:15.543992 16284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:21:15.545838 16284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:21:15.548434 16406 tablet_service.cc:1467] Processing CreateTablet for tablet 49e1ad2467d7474286867ed932189529 (DEFAULT_TABLE table=test-table [id=ae05fa9e77d841e48c24c6a8bcc3c036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:15.548651 16406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49e1ad2467d7474286867ed932189529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:15.550208 16364 tablet_bootstrap.cc:492] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f: Bootstrap starting.
I20260927 11:21:15.550525 16364 tablet_bootstrap.cc:654] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:15.551509 16364 tablet_bootstrap.cc:492] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f: No bootstrap required, opened a new log
I20260927 11:21:15.551564 16364 ts_tablet_manager.cc:1397] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:15.551759 16364 raft_consensus.cc:348] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:15.551843 16364 raft_consensus.cc:374] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:15.551899 16364 raft_consensus.cc:729] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55f75218ef48b58e1b327c7fc9610f, State: Initialized, Role: FOLLOWER
I20260927 11:21:15.551959 16364 consensus_queue.cc:260] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:15.552029 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.547631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42891 (local address 127.0.0.1:57664)
0927 11:21:15.547678 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:15.547691 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:15.547737 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:15.548392 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:15.548392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:15.548394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:15.548425 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.548426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.549721 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.549722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.550803 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.550806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.551755 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.551756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.551830 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.551853 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 11:21:15.551904 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 11:21:15.551952 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":13}
I20260927 11:21:15.552069 16364 ts_tablet_manager.cc:1428] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:21:15.552232 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.547725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46129 (local address 127.0.0.1:58112)
0927 11:21:15.547880 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:15.547891 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:15.547951 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:15.548193 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:15.548194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:15.548197 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:15.548246 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.548248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.549535 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.549536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.550869 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.550871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.551975 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.551976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.552051 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.552055 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:21:15.552092 (+ 37us) client_negotiation.cc:240] Negotiation successful
0927 11:21:15.552140 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
I20260927 11:21:15.552254 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.548002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.0.98.2:42891)
0927 11:21:15.548179 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:15.548184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:15.548197 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:21:15.548240 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:15.548242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:15.548258 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:15.548388 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:15.548477 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.549617 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.550917 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.551647 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.551755 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.551818 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:15.552129 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 11:21:15.552173 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260927 11:21:15.552680 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.547844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.0.98.3:46129)
0927 11:21:15.548022 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:15.548028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:15.548042 (+ 14us) server_negotiation.cc:407] Connection header received
0927 11:21:15.548077 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:15.548078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:15.548092 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:15.548181 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:15.548305 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.549456 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.550940 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.551858 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.552437 (+ 579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.552513 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:15.552544 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 11:21:15.552585 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
I20260927 11:21:15.552953 16545 tablet_service.cc:1467] Processing CreateTablet for tablet 49e1ad2467d7474286867ed932189529 (DEFAULT_TABLE table=test-table [id=ae05fa9e77d841e48c24c6a8bcc3c036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:15.553292 16545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49e1ad2467d7474286867ed932189529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:15.553368 16677 tablet_service.cc:1467] Processing CreateTablet for tablet 49e1ad2467d7474286867ed932189529 (DEFAULT_TABLE table=test-table [id=ae05fa9e77d841e48c24c6a8bcc3c036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:15.553663 16677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49e1ad2467d7474286867ed932189529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:15.555476 16503 tablet_bootstrap.cc:492] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca: Bootstrap starting.
I20260927 11:21:15.556418 16503 tablet_bootstrap.cc:654] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:15.556674 16503 log.cc:824] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:15.556995 16754 tablet_bootstrap.cc:492] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6: Bootstrap starting.
I20260927 11:21:15.557475 16503 tablet_bootstrap.cc:492] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca: No bootstrap required, opened a new log
I20260927 11:21:15.557528 16503 ts_tablet_manager.cc:1397] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:21:15.557629 16754 tablet_bootstrap.cc:654] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:15.558121 16754 log.cc:824] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:15.558820 16754 tablet_bootstrap.cc:492] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6: No bootstrap required, opened a new log
I20260927 11:21:15.558921 16754 ts_tablet_manager.cc:1397] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 11:21:15.559468 16503 raft_consensus.cc:348] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:15.559653 16503 raft_consensus.cc:374] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:15.559737 16503 raft_consensus.cc:729] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7dac5469b545d2bb2dbdaaf68293ca, State: Initialized, Role: FOLLOWER
I20260927 11:21:15.559870 16503 consensus_queue.cc:260] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:15.560173 16611 heartbeater.cc:503] Master 127.0.98.62:41273 was elected leader, sending a full tablet report...
I20260927 11:21:15.560433 16503 ts_tablet_manager.cc:1428] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:21:15.560695 16754 raft_consensus.cc:348] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:15.560880 16754 raft_consensus.cc:374] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:15.560907 16754 raft_consensus.cc:729] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c5aa0fe737498982d260dd4bc164d6, State: Initialized, Role: FOLLOWER
I20260927 11:21:15.561115 16754 consensus_queue.cc:260] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:15.561419 16754 ts_tablet_manager.cc:1428] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 11:21:15.561517 16743 heartbeater.cc:503] Master 127.0.98.62:41273 was elected leader, sending a full tablet report...
W20260927 11:21:15.632476 16612 tablet.cc:2369] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:21:15.663647 16757 raft_consensus.cc:482] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:21:15.663811 16757 raft_consensus.cc:504] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:15.664129 16757 leader_election.cc:288] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891), 9f55f75218ef48b58e1b327c7fc9610f (127.0.98.1:38241)
I20260927 11:21:15.668103 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.664583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53313 (local address 127.0.98.1:38241)
0927 11:21:15.664637 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:15.664639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:15.664690 (+ 51us) server_negotiation.cc:407] Connection header received
0927 11:21:15.664756 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:15.664757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:15.664779 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:15.664865 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:15.664998 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.665742 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.667175 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.667805 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.667871 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.667942 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:15.668002 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 11:21:15.668038 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 11:21:15.668103 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.664429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59897 (local address 127.0.98.2:42891)
0927 11:21:15.664483 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:15.664485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:15.664520 (+ 35us) server_negotiation.cc:407] Connection header received
0927 11:21:15.664548 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:15.664549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:15.664563 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:15.664612 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:15.664820 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.665620 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.667176 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.667725 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.667803 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.667814 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:15.667987 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 11:21:15.668032 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 11:21:15.668110 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.664326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42891 (local address 127.0.98.3:59897)
0927 11:21:15.664387 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:15.664401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:15.664478 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:15.664643 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:15.664644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:15.664646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:15.664748 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.664751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.665704 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.665705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.667084 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.667087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.667805 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.667806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.667902 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.667906 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:21:15.667969 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 11:21:15.668015 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":79}
I20260927 11:21:15.668119 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.664450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38241 (local address 127.0.98.3:53313)
0927 11:21:15.664627 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:15.664637 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:15.664691 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:15.664874 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:15.664874 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:15.664876 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:15.664929 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.664930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.665814 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.665815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.667084 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:15.667087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.667880 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.667881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:15.667945 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.667948 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:21:15.667995 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 11:21:15.668021 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260927 11:21:15.668475 16426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49e1ad2467d7474286867ed932189529" candidate_uuid: "61c5aa0fe737498982d260dd4bc164d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55f75218ef48b58e1b327c7fc9610f" is_pre_election: true
I20260927 11:21:15.668576 16565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49e1ad2467d7474286867ed932189529" candidate_uuid: "61c5aa0fe737498982d260dd4bc164d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" is_pre_election: true
I20260927 11:21:15.668651 16426 raft_consensus.cc:2454] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61c5aa0fe737498982d260dd4bc164d6 in term 0.
I20260927 11:21:15.668704 16565 raft_consensus.cc:2454] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61c5aa0fe737498982d260dd4bc164d6 in term 0.
I20260927 11:21:15.668926 16631 leader_election.cc:302] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61c5aa0fe737498982d260dd4bc164d6, 6d7dac5469b545d2bb2dbdaaf68293ca; no voters:
I20260927 11:21:15.669088 16757 raft_consensus.cc:2781] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:21:15.669169 16757 raft_consensus.cc:482] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:21:15.669236 16757 raft_consensus.cc:3037] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:15.670027 16757 raft_consensus.cc:504] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:15.670153 16757 leader_election.cc:288] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [CANDIDATE]: Term 1 election: Requested vote from peers 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891), 9f55f75218ef48b58e1b327c7fc9610f (127.0.98.1:38241)
I20260927 11:21:15.670331 16565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49e1ad2467d7474286867ed932189529" candidate_uuid: "61c5aa0fe737498982d260dd4bc164d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca"
I20260927 11:21:15.670393 16426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49e1ad2467d7474286867ed932189529" candidate_uuid: "61c5aa0fe737498982d260dd4bc164d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55f75218ef48b58e1b327c7fc9610f"
I20260927 11:21:15.670423 16565 raft_consensus.cc:3037] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:15.670467 16426 raft_consensus.cc:3037] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:15.671301 16565 raft_consensus.cc:2454] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61c5aa0fe737498982d260dd4bc164d6 in term 1.
I20260927 11:21:15.671301 16426 raft_consensus.cc:2454] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61c5aa0fe737498982d260dd4bc164d6 in term 1.
I20260927 11:21:15.671476 16631 leader_election.cc:302] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61c5aa0fe737498982d260dd4bc164d6, 9f55f75218ef48b58e1b327c7fc9610f; no voters:
I20260927 11:21:15.671573 16757 raft_consensus.cc:2781] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:21:15.671716 16757 raft_consensus.cc:686] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [term 1 LEADER]: Becoming Leader. State: Replica: 61c5aa0fe737498982d260dd4bc164d6, State: Running, Role: LEADER
I20260927 11:21:15.671813 16757 consensus_queue.cc:237] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:15.672647 16284 catalog_manager.cc:5495] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "61c5aa0fe737498982d260dd4bc164d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } health_report { overall_health: UNKNOWN } } }
I20260927 11:21:15.696290 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.694162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.0.98.3:46129)
0927 11:21:15.694219 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:15.694221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:15.694231 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:21:15.694277 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:15.694278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:15.694297 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:15.694364 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:21:15.694490 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.694966 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.695649 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:15.695830 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:15.695894 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:15.696003 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:21:15.696102 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:21:15.696163 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:15.696195 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 11:21:15.696211 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 11:21:15.698169 16565 raft_consensus.cc:1261] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 1 FOLLOWER]: Refusing update from remote peer 61c5aa0fe737498982d260dd4bc164d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:15.698179 16426 raft_consensus.cc:1261] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Refusing update from remote peer 61c5aa0fe737498982d260dd4bc164d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:15.698525 16759 consensus_queue.cc:1035] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:15.698668 16757 consensus_queue.cc:1035] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:15.701166 16763 mvcc.cc:205] Tried to move back new op lower bound from 7333921078056923136 to 7333921077952397312. Current Snapshot: MvccSnapshot[applied={T|T < 7333921078056923136}]
I20260927 11:21:15.703498 16284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:15.703593 16284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:15.704268 16284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Sending DeleteTablet for 3 replicas of tablet 49e1ad2467d7474286867ed932189529
I20260927 11:21:15.704584 16545 tablet_service.cc:1514] Processing DeleteTablet for tablet 49e1ad2467d7474286867ed932189529 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57664
I20260927 11:21:15.704593 16677 tablet_service.cc:1514] Processing DeleteTablet for tablet 49e1ad2467d7474286867ed932189529 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58112
I20260927 11:21:15.704622 16406 tablet_service.cc:1514] Processing DeleteTablet for tablet 49e1ad2467d7474286867ed932189529 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45466
I20260927 11:21:15.704701 16481 tablet_replica.cc:330] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f: stopping tablet replica
I20260927 11:21:15.704748 16481 raft_consensus.cc:2229] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:21:15.704800 16481 raft_consensus.cc:2258] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:15.704957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 16322
I20260927 11:21:15.705701 16772 tablet_replica.cc:330] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca: stopping tablet replica
I20260927 11:21:15.705818 16772 raft_consensus.cc:2229] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:21:15.705932 16772 raft_consensus.cc:2258] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:15.706116 16773 tablet_replica.cc:330] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6: stopping tablet replica
I20260927 11:21:15.706207 16773 raft_consensus.cc:2229] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:21:15.706328 16773 raft_consensus.cc:2258] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:15.706696 16772 ts_tablet_manager.cc:1905] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:15.706904 16773 ts_tablet_manager.cc:1905] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:15.708750 16772 ts_tablet_manager.cc:1918] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:15.708869 16772 log.cc:1196] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/49e1ad2467d7474286867ed932189529
I20260927 11:21:15.708966 16773 ts_tablet_manager.cc:1918] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:15.709101 16773 log.cc:1196] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/49e1ad2467d7474286867ed932189529
I20260927 11:21:15.709378 16772 ts_tablet_manager.cc:1939] T 49e1ad2467d7474286867ed932189529 P 6d7dac5469b545d2bb2dbdaaf68293ca: Deleting consensus metadata
I20260927 11:21:15.709396 16773 ts_tablet_manager.cc:1939] T 49e1ad2467d7474286867ed932189529 P 61c5aa0fe737498982d260dd4bc164d6: Deleting consensus metadata
I20260927 11:21:15.709789 16269 catalog_manager.cc:4841] TS 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129): tablet 49e1ad2467d7474286867ed932189529 (table test-table [id=ae05fa9e77d841e48c24c6a8bcc3c036]) successfully deleted
W20260927 11:21:15.711855 16269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38241 (error 108)
I20260927 11:21:15.712082 16269 catalog_manager.cc:4841] TS 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891): tablet 49e1ad2467d7474286867ed932189529 (table test-table [id=ae05fa9e77d841e48c24c6a8bcc3c036]) successfully deleted
I20260927 11:21:15.712332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 16476
I20260927 11:21:15.712347 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:15.712225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38241 (local address 127.0.0.1:37762)
0927 11:21:15.712267 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:15.712311 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260927 11:21:15.713016 16269 catalog_manager.cc:4584] TS 9f55f75218ef48b58e1b327c7fc9610f (127.0.98.1:38241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 49e1ad2467d7474286867ed932189529: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38241: connect: Connection refused (error 111)
I20260927 11:21:15.717952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 16614
I20260927 11:21:15.724040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 16251
I20260927 11:21:35.730126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41273
--webserver_interface=127.0.98.62
--webserver_port=43855
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:35.813655 16775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:35.813865 16775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:35.813887 16775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:35.815161 16775 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 11:21:35.815205 16775 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 11:21:35.815219 16775 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 11:21:35.815230 16775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:21:35.815243 16775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:35.815258 16775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:35.815269 16775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:21:35.815281 16775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:21:35.816835 16775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:35.817126 16775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:35.817376 16775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:35.822659 16781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:35.822690 16780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:35.822712 16775 server_base.cc:1034] running on GCE node
W20260927 11:21:35.822690 16783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:35.823169 16775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:35.823472 16775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:35.824656 16775 hybrid_clock.cc:648] HybridClock initialized: now 1790508095824646 us; error 56 us; skew 500 ppm
Sep 27 11:21:35 dist-test-slave-lsvg krb5kdc[15704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:35.831049 16775 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:21:35.831535 16775 webserver.cc:460] Webserver started at http://127.0.98.62:43855/ using document root <none> and password file <none>
I20260927 11:21:35.831790 16775 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:35.831866 16775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:35.833308 16775 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:21:35.834035 16790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:35.834236 16775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:35.834347 16775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "66bae122a7154fa58a97062b3d5db0a9"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "b04c2a4f3d8dd4f8f01d8f5bfb0e4021"
server_key_iv: "08508527e82c5380d9e997368799e843"
server_key_version: "encryptionkey@0"
I20260927 11:21:35.834712 16775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:35.844142 16775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:35.846163 16775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:35.846299 16775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:35.850109 16775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41273
I20260927 11:21:35.850204 16842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41273 every 8 connection(s)
I20260927 11:21:35.850544 16775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:21:35.852028 16843 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:21:35.852609 16843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Bootstrap starting.
I20260927 11:21:35.855690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 16775
I20260927 11:21:35.856055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38241
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41273
--webserver_port=37943
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:21:35.861570 16843 log.cc:824] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:35.864109 16843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:21:35.864447 16843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Bootstrap complete.
I20260927 11:21:35.866797 16843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } }
I20260927 11:21:35.867051 16843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bae122a7154fa58a97062b3d5db0a9, State: Initialized, Role: FOLLOWER
I20260927 11:21:35.867262 16843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } }
I20260927 11:21:35.867362 16843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:21:35.867416 16843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:21:35.867465 16843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 2 FOLLOWER]: Advancing to term 3
I20260927 11:21:35.868443 16843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } }
I20260927 11:21:35.868563 16843 leader_election.cc:302] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66bae122a7154fa58a97062b3d5db0a9; no voters:
I20260927 11:21:35.868764 16843 leader_election.cc:288] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 11:21:35.868832 16847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 11:21:35.868999 16847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [term 3 LEADER]: Becoming Leader. State: Replica: 66bae122a7154fa58a97062b3d5db0a9, State: Running, Role: LEADER
I20260927 11:21:35.869029 16843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:21:35.869138 16847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } }
I20260927 11:21:35.869414 16847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66bae122a7154fa58a97062b3d5db0a9. Latest consensus state: current_term: 3 leader_uuid: "66bae122a7154fa58a97062b3d5db0a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } } }
I20260927 11:21:35.869485 16847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:21:35.869418 16848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "66bae122a7154fa58a97062b3d5db0a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bae122a7154fa58a97062b3d5db0a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41273 } } }
I20260927 11:21:35.869787 16848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:21:35.869660 16853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:21:35.870723 16853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:21:35.870829 16853 catalog_manager.cc:1259] Loaded cluster ID: 2dc46c4f6e1e4008965154ff80de1dd8
I20260927 11:21:35.870859 16853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:21:35.872225 16853 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:21:35.872408 16853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Loaded TSK: 0
I20260927 11:21:35.872457 16853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 66bae122a7154fa58a97062b3d5db0a9: Loaded TSK: 1
F20260927 11:21:35.872718 16853 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb20b7a89ad google::LogMessage::Fail() at ??:0
@ 0x7fb20b7ab88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb20b7a84de google::LogMessage::Flush() at ??:0
@ 0x7fb20b7a9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb20df15104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb20df12d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb2137021cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb21370248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb213702594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb2136c8c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb2136d2ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb20d6760d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb20d66b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb20e4cf6db start_thread at ??:0
@ 0x7fb20a2a661f clone at ??:0
W20260927 11:21:35.960319 16845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:35.960526 16845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:35.960551 16845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:35.961894 16845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:35.961937 16845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:35.961966 16845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:21:35.963472 16845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37943
--tserver_master_addrs=127.0.98.62:41273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:35.963713 16845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:35.963955 16845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:35.970144 16885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:35.970325 16887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:35.970144 16884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:35.970552 16845 server_base.cc:1034] running on GCE node
I20260927 11:21:35.970825 16845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:35.971096 16845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:35.972282 16845 hybrid_clock.cc:648] HybridClock initialized: now 1790508095972256 us; error 45 us; skew 500 ppm
Sep 27 11:21:35 dist-test-slave-lsvg krb5kdc[15704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:35.974958 16845 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:21:35.975469 16845 webserver.cc:460] Webserver started at http://127.0.98.1:37943/ using document root <none> and password file <none>
I20260927 11:21:35.975708 16845 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:35.975778 16845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:35.977286 16845 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:35.978103 16894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:35.978330 16845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:35.978439 16845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "9f55f75218ef48b58e1b327c7fc9610f"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "37ac57d20610953e3333ac578d6f6a0a"
server_key_iv: "a04f1190dbd854e30a44aea5a9b3aec7"
server_key_version: "encryptionkey@0"
I20260927 11:21:35.978822 16845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:36.024019 16845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:36.026604 16845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:36.026788 16845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:36.027050 16845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:36.027560 16901 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 11:21:36.029569 16845 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 11:21:36.029667 16845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 11:21:36.029718 16845 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 11:21:36.030426 16845 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 11:21:36.030486 16845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 11:21:36.030558 16901 tablet_bootstrap.cc:492] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f: Bootstrap starting.
I20260927 11:21:36.032886 16901 log.cc:824] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:36.036571 16901 tablet_bootstrap.cc:492] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:21:36.037091 16901 tablet_bootstrap.cc:492] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f: Bootstrap complete.
I20260927 11:21:36.037266 16901 ts_tablet_manager.cc:1397] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260927 11:21:36.037835 16845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38241
I20260927 11:21:36.038053 17009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38241 every 8 connection(s)
I20260927 11:21:36.038275 16845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:21:36.039223 16901 raft_consensus.cc:348] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:36.039578 16901 raft_consensus.cc:729] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55f75218ef48b58e1b327c7fc9610f, State: Initialized, Role: FOLLOWER
I20260927 11:21:36.039762 16901 consensus_queue.cc:260] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:36.040023 16901 ts_tablet_manager.cc:1428] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 11:21:36.042223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 16845
I20260927 11:21:36.042551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42891
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41273
--webserver_port=32853
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:36.100540 17012 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:21:36.038994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:50313)
0927 11:21:36.039222 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:36.039269 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:36.039422 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:36.100300 (+ 60878us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":119,"threads_started":1}
W20260927 11:21:36.100857 17010 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41273 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41273: Client connection negotiation failed: client connection to 127.0.98.62:41273: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 11:21:36.101292 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:36.101163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:36521)
0927 11:21:36.101222 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:36.101270 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 11:21:36.101753 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:36.101603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:53903)
0927 11:21:36.101697 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:36.101730 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 11:21:36.101908 17010 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 11:21:36.139847 17014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:36.140108 17014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:36.140151 17014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:36.141472 17014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:36.141536 17014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:36.141580 17014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:21:36.143103 17014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=32853
--tserver_master_addrs=127.0.98.62:41273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:36.143388 17014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:36.143635 17014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:36.148737 17019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:36.148746 17022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:36.148748 17020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:36.149089 17014 server_base.cc:1034] running on GCE node
I20260927 11:21:36.149324 17014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:36.149576 17014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:36.150789 17014 hybrid_clock.cc:648] HybridClock initialized: now 1790508096150763 us; error 31 us; skew 500 ppm
Sep 27 11:21:36 dist-test-slave-lsvg krb5kdc[15704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:36.153700 17014 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:21:36.154186 17014 webserver.cc:460] Webserver started at http://127.0.98.2:32853/ using document root <none> and password file <none>
I20260927 11:21:36.154435 17014 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:36.154505 17014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:36.155860 17014 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:36.156481 17029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:36.156714 17014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:21:36.156800 17014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "6d7dac5469b545d2bb2dbdaaf68293ca"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "f75223cf323f07c1f3003899cd62c683"
server_key_iv: "6e9fefe59e6ec099f94b7f22e60e980d"
server_key_version: "encryptionkey@0"
I20260927 11:21:36.157189 17014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:36.165036 17014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:36.167136 17014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:36.167274 17014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:36.167447 17014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:36.167747 17014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:36.167784 17014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:36.167807 17014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:36.167855 17014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:36.173947 17014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42891
I20260927 11:21:36.174032 17142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42891 every 8 connection(s)
I20260927 11:21:36.174384 17014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:21:36.175642 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:36.175041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:48377)
0927 11:21:36.175231 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:36.175299 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
W20260927 11:21:36.175884 17143 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41273 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41273: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
I20260927 11:21:36.176154 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:36.176071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:54011)
0927 11:21:36.176112 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:36.176137 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 11:21:36.176501 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:36.176450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:53797)
0927 11:21:36.176471 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:36.176489 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260927 11:21:36.176626 17143 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 11:21:36.178301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 17014
I20260927 11:21:36.178632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46129
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41273
--webserver_port=35329
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:36.261412 17146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:36.261685 17146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:36.261737 17146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:36.263119 17146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:36.263191 17146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:36.263238 17146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:21:36.264822 17146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35329
--tserver_master_addrs=127.0.98.62:41273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:36.265146 17146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:36.265409 17146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:36.270629 17154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:36.270748 17151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:36.270879 17146 server_base.cc:1034] running on GCE node
W20260927 11:21:36.270629 17152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:36.271217 17146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:36.271489 17146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:36.272665 17146 hybrid_clock.cc:648] HybridClock initialized: now 1790508096272638 us; error 35 us; skew 500 ppm
Sep 27 11:21:36 dist-test-slave-lsvg krb5kdc[15704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:36.275458 17146 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:21:36.275944 17146 webserver.cc:460] Webserver started at http://127.0.98.3:35329/ using document root <none> and password file <none>
I20260927 11:21:36.276229 17146 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:36.276310 17146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:36.277828 17146 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:21:36.278522 17161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:36.278699 17146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:21:36.278813 17146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "61c5aa0fe737498982d260dd4bc164d6"
format_stamp: "Formatted at 2026-09-27 11:20:54 on dist-test-slave-lsvg"
server_key: "be5b5e8442896e978afa2b524b027c65"
server_key_iv: "76ae0541309bea92ea1c55afaa6cc454"
server_key_version: "encryptionkey@0"
I20260927 11:21:36.279237 17146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:36.296156 17146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:36.298363 17146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:36.298557 17146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:36.298826 17146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:36.299185 17146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:36.299249 17146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:36.299297 17146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:36.299324 17146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:36.305847 17146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46129
I20260927 11:21:36.305902 17274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46129 every 8 connection(s)
I20260927 11:21:36.306281 17146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:21:36.307379 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:36.306953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:38053)
0927 11:21:36.307106 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:36.307212 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
W20260927 11:21:36.307785 17275 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41273 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41273: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
I20260927 11:21:36.308092 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:36.307981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:42925)
0927 11:21:36.308034 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:36.308066 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 11:21:36.308384 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:36.308294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:49781)
0927 11:21:36.308340 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:36.308364 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 11:21:36.308493 17275 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 11:21:36.313720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 17146
I20260927 11:21:37.102679 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:37.102318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:43687)
0927 11:21:37.102524 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:37.102594 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
I20260927 11:21:37.177270 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:37.176978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:36939)
0927 11:21:37.177175 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:37.177217 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":56,"threads_started":1}
I20260927 11:21:37.309016 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:37.308808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:55445)
0927 11:21:37.308926 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:37.308964 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260927 11:21:37.312881 17282 raft_consensus.cc:482] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:21:37.312983 17282 raft_consensus.cc:504] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:37.313390 17282 leader_election.cc:288] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891), 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129)
Sep 27 11:21:37 dist-test-slave-lsvg krb5kdc[15704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 11:21:37 dist-test-slave-lsvg krb5kdc[15704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 11:21:37.324751 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:37.313774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44417 (local address 127.0.98.3:46129)
0927 11:21:37.313912 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:37.313918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:37.313945 (+ 27us) server_negotiation.cc:407] Connection header received
0927 11:21:37.314497 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:37.314500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:37.314562 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:37.314712 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:37.315556 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:37.316413 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:37.317395 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:37.317906 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:37.318000 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:37.321351 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:37.321354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:37.321354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:37.321365 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:37.323058 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:37.323517 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:37.323519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:37.323519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:37.323602 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:37.323864 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:37.323865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:37.323866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:37.324123 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:37.324208 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:37.324522 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 11:21:37.324644 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":44,"threads_started":1}
I20260927 11:21:37.324750 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:37.313584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42891 (local address 127.0.98.1:49465)
0927 11:21:37.313640 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:37.313652 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:37.313733 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:37.314755 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:37.314758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:37.315051 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:37.315254 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:37.315260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:37.316431 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:37.316432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:37.317296 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:37.317298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:37.318105 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:37.318106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:37.318158 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:37.318899 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:37.318904 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:37.320703 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:37.322979 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:37.322986 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:37.322988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:37.323388 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:37.323683 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:37.323684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:37.323685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:37.323804 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:37.324219 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:37.324220 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:37.324338 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 11:21:37.324421 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 11:21:37.324637 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 11:21:37.324774 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:37.313735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49465 (local address 127.0.98.2:42891)
0927 11:21:37.313870 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:37.313882 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:37.313916 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:21:37.314497 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:37.314500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:37.314561 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:37.314725 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:37.315381 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:37.316308 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:37.317574 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:37.318011 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:37.318116 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:37.320823 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:37.320827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:37.320827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:37.320838 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:37.322876 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:37.323481 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:37.323484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:37.323485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:37.323603 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:37.323897 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:37.323898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:37.323898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:37.324125 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:37.324231 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:37.324500 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 11:21:37.324662 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260927 11:21:37.325507 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:37.313693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46129 (local address 127.0.98.1:44417)
0927 11:21:37.313806 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:37.313816 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:37.313867 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:37.314840 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:37.314841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:37.315377 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:37.315452 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:37.315454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:37.316509 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:37.316510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:37.317298 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:37.317301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:37.318010 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:37.318011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:37.318138 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:37.318757 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:37.318769 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:37.321217 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:37.323146 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:37.323147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:37.323148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:37.323412 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:37.323675 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:37.323677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:37.323677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:37.323799 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:37.324197 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:37.324205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:37.324339 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 11:21:37.324451 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 11:21:37.324719 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":175,"thread_start_us":50,"threads_started":1}
W20260927 11:21:37.325783 16896 leader_election.cc:341] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891): Not found: Tablet not found: 49e1ad2467d7474286867ed932189529
W20260927 11:21:37.326256 16896 leader_election.cc:341] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129): Not found: Tablet not found: 49e1ad2467d7474286867ed932189529
I20260927 11:21:37.326316 16896 leader_election.cc:302] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f55f75218ef48b58e1b327c7fc9610f; no voters: 61c5aa0fe737498982d260dd4bc164d6, 6d7dac5469b545d2bb2dbdaaf68293ca
I20260927 11:21:37.326455 17282 raft_consensus.cc:2726] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:21:38.103412 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:38.103196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:35589)
0927 11:21:38.103331 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:38.103364 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
I20260927 11:21:38.177989 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:38.177781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:50911)
0927 11:21:38.177899 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:38.177935 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260927 11:21:38.309782 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:38.309522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:35407)
0927 11:21:38.309680 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:38.309741 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260927 11:21:39.104400 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:39.104152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:36261)
0927 11:21:39.104305 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:39.104351 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260927 11:21:39.178712 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:39.178468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:41613)
0927 11:21:39.178622 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:39.178668 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
I20260927 11:21:39.310511 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:39.310272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:48707)
0927 11:21:39.310426 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:39.310467 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
I20260927 11:21:39.351855 17292 raft_consensus.cc:482] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:21:39.351956 17292 raft_consensus.cc:504] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:39.352165 17292 leader_election.cc:288] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891), 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129)
W20260927 11:21:39.352605 16896 leader_election.cc:341] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891): Not found: Tablet not found: 49e1ad2467d7474286867ed932189529
W20260927 11:21:39.352700 16896 leader_election.cc:341] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129): Not found: Tablet not found: 49e1ad2467d7474286867ed932189529
I20260927 11:21:39.352739 16896 leader_election.cc:302] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f55f75218ef48b58e1b327c7fc9610f; no voters: 61c5aa0fe737498982d260dd4bc164d6, 6d7dac5469b545d2bb2dbdaaf68293ca
I20260927 11:21:39.352815 17292 raft_consensus.cc:2726] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:21:40.105284 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:40.104992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:45423)
0927 11:21:40.105181 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:40.105226 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":129,"threads_started":1}
I20260927 11:21:40.179380 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:40.179153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:55515)
0927 11:21:40.179294 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:40.179335 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260927 11:21:40.311240 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:40.311020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:42281)
0927 11:21:40.311157 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:40.311199 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260927 11:21:40.956754 17296 raft_consensus.cc:482] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:21:40.956980 17296 raft_consensus.cc:504] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:40.957239 17296 leader_election.cc:288] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891), 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129)
W20260927 11:21:40.957712 16896 leader_election.cc:341] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891): Not found: Tablet not found: 49e1ad2467d7474286867ed932189529
W20260927 11:21:40.957845 16896 leader_election.cc:341] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129): Not found: Tablet not found: 49e1ad2467d7474286867ed932189529
I20260927 11:21:40.957885 16896 leader_election.cc:302] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f55f75218ef48b58e1b327c7fc9610f; no voters: 61c5aa0fe737498982d260dd4bc164d6, 6d7dac5469b545d2bb2dbdaaf68293ca
I20260927 11:21:40.957983 17296 raft_consensus.cc:2726] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:21:41.106068 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:41.105838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:42725)
0927 11:21:41.105984 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:41.106020 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260927 11:21:41.180063 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:41.179849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:59887)
0927 11:21:41.179983 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:41.180021 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260927 11:21:41.311977 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:41.311760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:58293)
0927 11:21:41.311893 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:41.311933 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260927 11:21:42.106922 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:42.106657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:34397)
0927 11:21:42.106820 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:42.106872 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260927 11:21:42.180837 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:42.180594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:34205)
0927 11:21:42.180738 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:42.180789 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260927 11:21:42.312752 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:42.312483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:48099)
0927 11:21:42.312659 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:42.312704 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
I20260927 11:21:43.107717 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:43.107473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:56755)
0927 11:21:43.107626 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:43.107669 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260927 11:21:43.181710 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:43.181470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:33329)
0927 11:21:43.181613 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:43.181664 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260927 11:21:43.214758 17305 raft_consensus.cc:482] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:21:43.214895 17305 raft_consensus.cc:504] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:43.215085 17305 leader_election.cc:288] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891), 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129)
W20260927 11:21:43.215543 16896 leader_election.cc:341] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891): Not found: Tablet not found: 49e1ad2467d7474286867ed932189529
W20260927 11:21:43.215615 16896 leader_election.cc:341] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129): Not found: Tablet not found: 49e1ad2467d7474286867ed932189529
I20260927 11:21:43.215631 16896 leader_election.cc:302] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f55f75218ef48b58e1b327c7fc9610f; no voters: 61c5aa0fe737498982d260dd4bc164d6, 6d7dac5469b545d2bb2dbdaaf68293ca
I20260927 11:21:43.215749 17305 raft_consensus.cc:2726] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:21:43.313476 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:43.313223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:33303)
0927 11:21:43.313398 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:43.313431 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":107,"threads_started":1}
I20260927 11:21:44.108494 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:44.108245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:55465)
0927 11:21:44.108404 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:44.108445 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260927 11:21:44.182308 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:44.182113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:37123)
0927 11:21:44.182228 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:44.182265 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260927 11:21:44.314224 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:44.313984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:49685)
0927 11:21:44.314135 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:44.314179 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260927 11:21:45.109512 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:45.109224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:40527)
0927 11:21:45.109401 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:45.109451 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260927 11:21:45.183099 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:45.182833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:38949)
0927 11:21:45.182986 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:45.183051 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
I20260927 11:21:45.315016 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:45.314765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:40853)
0927 11:21:45.314921 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:45.314968 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260927 11:21:45.529492 17313 raft_consensus.cc:482] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:21:45.529630 17313 raft_consensus.cc:504] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7dac5469b545d2bb2dbdaaf68293ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } } peers { permanent_uuid: "61c5aa0fe737498982d260dd4bc164d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46129 } } peers { permanent_uuid: "9f55f75218ef48b58e1b327c7fc9610f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } }
I20260927 11:21:45.529824 17313 leader_election.cc:288] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891), 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129)
W20260927 11:21:45.530319 16896 leader_election.cc:341] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61c5aa0fe737498982d260dd4bc164d6 (127.0.98.3:46129): Not found: Tablet not found: 49e1ad2467d7474286867ed932189529
W20260927 11:21:45.530448 16896 leader_election.cc:341] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6d7dac5469b545d2bb2dbdaaf68293ca (127.0.98.2:42891): Not found: Tablet not found: 49e1ad2467d7474286867ed932189529
I20260927 11:21:45.530491 16896 leader_election.cc:302] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f55f75218ef48b58e1b327c7fc9610f; no voters: 61c5aa0fe737498982d260dd4bc164d6, 6d7dac5469b545d2bb2dbdaaf68293ca
I20260927 11:21:45.530560 17313 raft_consensus.cc:2726] T 49e1ad2467d7474286867ed932189529 P 9f55f75218ef48b58e1b327c7fc9610f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:21:46.110289 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:46.110063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:47583)
0927 11:21:46.110195 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:46.110239 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260927 11:21:46.183796 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:46.183564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:59855)
0927 11:21:46.183709 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:46.183752 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260927 11:21:46.315760 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:46.315516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:41247)
0927 11:21:46.315676 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:46.315715 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260927 11:21:47.111002 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:47.110755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:52931)
0927 11:21:47.110911 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:47.110954 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260927 11:21:47.184494 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:47.184262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:59051)
0927 11:21:47.184398 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:47.184445 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
I20260927 11:21:47.316682 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:47.316424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:58221)
0927 11:21:47.316581 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:47.316613 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260927 11:21:48.111815 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:48.111506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:42729)
0927 11:21:48.111724 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:48.111767 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":59,"threads_started":1}
I20260927 11:21:48.185297 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:48.185018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:58841)
0927 11:21:48.185206 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:48.185251 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":123,"threads_started":1}
I20260927 11:21:48.317374 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:48.317185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:53377)
0927 11:21:48.317297 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:48.317331 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260927 11:21:49.112707 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:49.112433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:53507)
0927 11:21:49.112607 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:49.112658 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
I20260927 11:21:49.186049 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:49.185770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:42951)
0927 11:21:49.185934 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:49.186000 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":61,"threads_started":1}
I20260927 11:21:49.318272 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:49.318003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:56067)
0927 11:21:49.318171 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:49.318224 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260927 11:21:50.113664 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:50.113442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:53743)
0927 11:21:50.113579 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:50.113617 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260927 11:21:50.186844 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:50.186582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:56905)
0927 11:21:50.186757 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:50.186796 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260927 11:21:50.319082 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:50.318817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:57497)
0927 11:21:50.318982 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:50.319037 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
I20260927 11:21:51.114642 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:51.114412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.1:50009)
0927 11:21:51.114547 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:51.114591 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260927 11:21:51.187649 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:51.187417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.2:41523)
0927 11:21:51.187565 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:51.187604 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260927 11:21:51.319307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 16845
I20260927 11:21:51.321041 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:51.319664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41273 (local address 127.0.98.3:37727)
0927 11:21:51.320956 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:51.320998 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1265,"thread_start_us":55,"threads_started":1}
I20260927 11:21:51.324357 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 17014
I20260927 11:21:51.328621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 17146
2026-09-27T11:21:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58418 ms)
[----------] 30 tests from SecurityITest (233187 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:21:51 dist-test-slave-lsvg krb5kdc[17334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:21:51 dist-test-slave-lsvg krb5kdc[17334](info): set up 2 sockets
Sep 27 11:21:51 dist-test-slave-lsvg krb5kdc[17334](info): commencing operation
krb5kdc: starting...
W20260927 11:21:52.377244 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:21:52 dist-test-slave-lsvg krb5kdc[17334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:21:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:21:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:21:52.504047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34443
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:52.588158 17347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:52.588416 17347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:52.588451 17347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:52.589762 17347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:21:52.589824 17347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:52.589847 17347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:52.589879 17347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:21:52.589905 17347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:21:52.591378 17347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:52.591656 17347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:52.591879 17347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:52.596619 17355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:52.596757 17353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:52.596702 17352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:52.596833 17347 server_base.cc:1034] running on GCE node
I20260927 11:21:52.597188 17347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:52.597432 17347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:52.598600 17347 hybrid_clock.cc:648] HybridClock initialized: now 1790508112598589 us; error 32 us; skew 500 ppm
Sep 27 11:21:52 dist-test-slave-lsvg krb5kdc[17334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:52.601384 17347 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:21:52.601879 17347 webserver.cc:460] Webserver started at http://127.0.98.62:36791/ using document root <none> and password file <none>
I20260927 11:21:52.602101 17347 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:52.602169 17347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:52.602320 17347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:52.603358 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "2605df2874364c968930ce7c42ebd7ec"
format_stamp: "Formatted at 2026-09-27 11:21:52 on dist-test-slave-lsvg"
server_key: "891d07cca098b708b42b75f484674b3f"
server_key_iv: "0868585df9654b596187ccadf4dbe38a"
server_key_version: "encryptionkey@0"
I20260927 11:21:52.603724 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "2605df2874364c968930ce7c42ebd7ec"
format_stamp: "Formatted at 2026-09-27 11:21:52 on dist-test-slave-lsvg"
server_key: "891d07cca098b708b42b75f484674b3f"
server_key_iv: "0868585df9654b596187ccadf4dbe38a"
server_key_version: "encryptionkey@0"
I20260927 11:21:52.605088 17347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:21:52.605886 17362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:52.606177 17347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:52.606280 17347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "2605df2874364c968930ce7c42ebd7ec"
format_stamp: "Formatted at 2026-09-27 11:21:52 on dist-test-slave-lsvg"
server_key: "891d07cca098b708b42b75f484674b3f"
server_key_iv: "0868585df9654b596187ccadf4dbe38a"
server_key_version: "encryptionkey@0"
I20260927 11:21:52.606359 17347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:52.624042 17347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:52.626425 17347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:52.626616 17347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:52.630786 17347 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34443
I20260927 11:21:52.630820 17414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34443 every 8 connection(s)
I20260927 11:21:52.631196 17347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:21:52.632051 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:52.634399 17415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec: Bootstrap starting.
I20260927 11:21:52.635000 17415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:52.635308 17415 log.cc:824] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:52.636041 17415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec: No bootstrap required, opened a new log
I20260927 11:21:52.637691 17415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2605df2874364c968930ce7c42ebd7ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34443 } }
I20260927 11:21:52.637835 17415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:52.637857 17415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2605df2874364c968930ce7c42ebd7ec, State: Initialized, Role: FOLLOWER
I20260927 11:21:52.638023 17415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2605df2874364c968930ce7c42ebd7ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34443 } }
I20260927 11:21:52.638099 17415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:21:52.638141 17415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:21:52.638211 17415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:52.638747 17415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2605df2874364c968930ce7c42ebd7ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34443 } }
I20260927 11:21:52.638857 17415 leader_election.cc:302] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2605df2874364c968930ce7c42ebd7ec; no voters:
I20260927 11:21:52.639047 17415 leader_election.cc:288] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:21:52.639093 17418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:21:52.639324 17418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [term 1 LEADER]: Becoming Leader. State: Replica: 2605df2874364c968930ce7c42ebd7ec, State: Running, Role: LEADER
I20260927 11:21:52.639323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 17347
I20260927 11:21:52.639423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:21:52.639344 17415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:21:52.639467 17418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2605df2874364c968930ce7c42ebd7ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34443 } }
I20260927 11:21:52.639654 392 external_mini_cluster.cc:1428] Setting key a3372de68ab29d229e015fdeae4d6115
I20260927 11:21:52.639845 17420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2605df2874364c968930ce7c42ebd7ec. Latest consensus state: current_term: 1 leader_uuid: "2605df2874364c968930ce7c42ebd7ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2605df2874364c968930ce7c42ebd7ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34443 } } }
I20260927 11:21:52.639982 17419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2605df2874364c968930ce7c42ebd7ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2605df2874364c968930ce7c42ebd7ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34443 } } }
I20260927 11:21:52.640002 17420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [sys.catalog]: This master's current role is: LEADER
I20260927 11:21:52.640045 17419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec [sys.catalog]: This master's current role is: LEADER
I20260927 11:21:52.640259 17427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:21:52.640909 17427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:21:52.642956 17427 catalog_manager.cc:1347] Generated new cluster ID: e5480a4be92542f5b71ce01a998eb894
I20260927 11:21:52.643013 17427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:21:52 dist-test-slave-lsvg krb5kdc[17334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:21:52.650753 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:52.640359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.0.98.62:34443)
0927 11:21:52.641375 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:52.641383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:52.641408 (+ 25us) server_negotiation.cc:407] Connection header received
0927 11:21:52.641608 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:52.641616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:52.641677 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:52.641825 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:52.642620 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:52.643650 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:52.644461 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:52.644883 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:52.645002 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:52.647335 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:52.647339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:52.647339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:52.647359 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:52.649001 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:52.649444 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:52.649447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:52.649448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:52.649551 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:52.649778 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:52.649779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:52.649779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:52.650024 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:52.650187 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:52.650446 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 11:21:52.650589 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":80,"threads_started":1}
I20260927 11:21:52.674688 17427 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:21:52.675241 17427 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:21:52.681406 17427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec: Generated new TSK 0
I20260927 11:21:52.681614 17427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:21:52.708714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34443
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:52.812562 17441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:52.812780 17441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:52.812810 17441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:52.814131 17441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:52.814182 17441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:52.814213 17441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:21:52.815740 17441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:52.816010 17441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:52.816269 17441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:52.822346 17446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:52.822463 17441 server_base.cc:1034] running on GCE node
W20260927 11:21:52.822326 17449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:52.822326 17447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:52.822857 17441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:52.823154 17441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:52.824334 17441 hybrid_clock.cc:648] HybridClock initialized: now 1790508112824323 us; error 34 us; skew 500 ppm
Sep 27 11:21:52 dist-test-slave-lsvg krb5kdc[17334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:52.826963 17441 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:21:52.827456 17441 webserver.cc:460] Webserver started at http://127.0.98.1:36953/ using document root <none> and password file <none>
I20260927 11:21:52.827710 17441 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:52.827780 17441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:52.827936 17441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:52.829006 17441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "f610bfe38d064de99db3b8776a792a88"
format_stamp: "Formatted at 2026-09-27 11:21:52 on dist-test-slave-lsvg"
server_key: "ab2f4996d7a87a3b122b16e54c8be7e7"
server_key_iv: "426edcd1a377ec53bb503b58a395c825"
server_key_version: "encryptionkey@0"
I20260927 11:21:52.829434 17441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f610bfe38d064de99db3b8776a792a88"
format_stamp: "Formatted at 2026-09-27 11:21:52 on dist-test-slave-lsvg"
server_key: "ab2f4996d7a87a3b122b16e54c8be7e7"
server_key_iv: "426edcd1a377ec53bb503b58a395c825"
server_key_version: "encryptionkey@0"
I20260927 11:21:52.830760 17441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:52.831511 17456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:52.831677 17441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:21:52.831775 17441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "f610bfe38d064de99db3b8776a792a88"
format_stamp: "Formatted at 2026-09-27 11:21:52 on dist-test-slave-lsvg"
server_key: "ab2f4996d7a87a3b122b16e54c8be7e7"
server_key_iv: "426edcd1a377ec53bb503b58a395c825"
server_key_version: "encryptionkey@0"
I20260927 11:21:52.831863 17441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:52.847635 17441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:52.849805 17441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:52.849970 17441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:52.850211 17441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:52.850579 17441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:52.850626 17441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:52.850668 17441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:52.850695 17441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:52.856688 17441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35625
I20260927 11:21:52.856732 17569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35625 every 8 connection(s)
I20260927 11:21:52.857113 17441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:21:52 dist-test-slave-lsvg krb5kdc[17334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:21:52.864627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 17441
I20260927 11:21:52.864755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:21:52.864971 392 external_mini_cluster.cc:1428] Setting key 810563bcfd82501138013ccf66a1cdcd
I20260927 11:21:52.866647 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:52.857839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34141 (local address 127.0.98.62:34443)
0927 11:21:52.857903 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:52.857905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:52.858155 (+ 250us) server_negotiation.cc:407] Connection header received
0927 11:21:52.859091 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:52.859093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:52.859106 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:52.859164 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:52.859754 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:52.860254 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:52.861265 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:52.861464 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:52.861547 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:52.863601 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:52.863602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:52.863602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:52.863608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:52.865157 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:52.865640 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:52.865641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:52.865641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:52.865707 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:52.866012 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:52.866013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:52.866013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:52.866116 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:52.866165 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:52.866503 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 11:21:52.866586 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:21:52.866938 17572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:52.857824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34443 (local address 127.0.98.1:34141)
0927 11:21:52.858019 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:52.858055 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:52.858979 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:52.859222 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:52.859225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:52.859479 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:52.859680 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:52.859684 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:52.860336 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:52.860337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:52.861192 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:52.861195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:52.861579 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:52.861580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:52.861682 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:52.862157 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:52.862164 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:52.863501 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:52.865243 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:52.865248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:52.865250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:52.865530 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:52.865781 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:52.865782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:52.865782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:52.865931 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:52.866181 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:52.866187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:52.866364 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 11:21:52.866484 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 11:21:52.866663 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
I20260927 11:21:52.867440 17570 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34443
I20260927 11:21:52.867605 17570 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:52.867807 17570 heartbeater.cc:511] Master 127.0.98.62:34443 requested a full tablet report, sending...
I20260927 11:21:52.868312 17379 ts_manager.cc:194] Registered new tserver with Master: f610bfe38d064de99db3b8776a792a88 (127.0.98.1:35625)
I20260927 11:21:52.868920 17379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34141
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:21:52.893011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34443
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:52.974557 17575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:52.974767 17575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:52.974790 17575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:52.975891 17575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:52.975924 17575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:52.975950 17575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:21:52.977524 17575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:52.977710 17575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:52.977885 17575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:52.983017 17580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:52.983053 17581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:52.983044 17583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:52.983212 17575 server_base.cc:1034] running on GCE node
I20260927 11:21:52.983431 17575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:52.983695 17575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:52.984862 17575 hybrid_clock.cc:648] HybridClock initialized: now 1790508112984850 us; error 43 us; skew 500 ppm
Sep 27 11:21:52 dist-test-slave-lsvg krb5kdc[17334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:52.987633 17575 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:21:52.988065 17575 webserver.cc:460] Webserver started at http://127.0.98.2:45365/ using document root <none> and password file <none>
I20260927 11:21:52.988300 17575 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:52.988370 17575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:52.988524 17575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:52.989658 17575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "041083b65faa4f7184e69bf5dd082de3"
format_stamp: "Formatted at 2026-09-27 11:21:52 on dist-test-slave-lsvg"
server_key: "afa3a8f55649c34cb48b1d8a55d0bdac"
server_key_iv: "33612351b311858f9322ad853ca064da"
server_key_version: "encryptionkey@0"
I20260927 11:21:52.990026 17575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "041083b65faa4f7184e69bf5dd082de3"
format_stamp: "Formatted at 2026-09-27 11:21:52 on dist-test-slave-lsvg"
server_key: "afa3a8f55649c34cb48b1d8a55d0bdac"
server_key_iv: "33612351b311858f9322ad853ca064da"
server_key_version: "encryptionkey@0"
I20260927 11:21:52.991480 17575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:21:52.992197 17590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:52.992372 17575 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:21:52.992472 17575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "041083b65faa4f7184e69bf5dd082de3"
format_stamp: "Formatted at 2026-09-27 11:21:52 on dist-test-slave-lsvg"
server_key: "afa3a8f55649c34cb48b1d8a55d0bdac"
server_key_iv: "33612351b311858f9322ad853ca064da"
server_key_version: "encryptionkey@0"
I20260927 11:21:52.992558 17575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:53.005705 17575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:53.008144 17575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:53.008270 17575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:53.008455 17575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:53.008786 17575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:53.008821 17575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.008867 17575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:53.008891 17575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.015184 17575 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43997
I20260927 11:21:53.015224 17703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43997 every 8 connection(s)
I20260927 11:21:53.015582 17575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:21:53.017980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 17575
I20260927 11:21:53.018105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:21:53.018338 392 external_mini_cluster.cc:1428] Setting key 858982df7c63e9669ea137a07ffa9786
Sep 27 11:21:53 dist-test-slave-lsvg krb5kdc[17334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 11:21:53.027007 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.016397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37755 (local address 127.0.98.62:34443)
0927 11:21:53.016461 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.016463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.016666 (+ 203us) server_negotiation.cc:407] Connection header received
0927 11:21:53.017699 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.017701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.017720 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.017776 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:53.018707 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.019286 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.020816 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.021159 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.021240 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.023858 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:53.023860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:53.023861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:53.023869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:53.025463 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:53.025907 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:53.025908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:53.025909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:53.025984 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:53.026269 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:53.026270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:53.026271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:53.026395 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:53.026473 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.026819 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.026910 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:21:53.027225 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.016379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34443 (local address 127.0.98.2:37755)
0927 11:21:53.016535 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.016573 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:53.017598 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:53.017835 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:53.017839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:53.018263 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:53.018549 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.018556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.019376 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.019377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.020734 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.020737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.021242 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.021243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.021344 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.021982 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:53.021992 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:53.023751 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:53.025570 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:53.025578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:53.025581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:53.025826 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:53.026069 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:53.026070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:53.026070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:53.026199 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:53.026465 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:53.026472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:53.026657 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 11:21:53.026799 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 11:21:53.026969 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260927 11:21:53.027766 17704 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34443
I20260927 11:21:53.027896 17704 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:53.028083 17704 heartbeater.cc:511] Master 127.0.98.62:34443 requested a full tablet report, sending...
I20260927 11:21:53.028501 17379 ts_manager.cc:194] Registered new tserver with Master: 041083b65faa4f7184e69bf5dd082de3 (127.0.98.2:43997)
I20260927 11:21:53.028854 17379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:21:53.045028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34443
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:53.127099 17709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:53.127312 17709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:53.127336 17709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:53.128595 17709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:53.128643 17709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:53.128671 17709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:21:53.130232 17709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:53.130440 17709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:53.130631 17709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:53.135788 17715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:53.135874 17717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:53.135923 17709 server_base.cc:1034] running on GCE node
W20260927 11:21:53.135788 17714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:53.136325 17709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:53.136549 17709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:53.137686 17709 hybrid_clock.cc:648] HybridClock initialized: now 1790508113137678 us; error 41 us; skew 500 ppm
Sep 27 11:21:53 dist-test-slave-lsvg krb5kdc[17334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:53.140717 17709 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:21:53.141345 17709 webserver.cc:460] Webserver started at http://127.0.98.3:34595/ using document root <none> and password file <none>
I20260927 11:21:53.141640 17709 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:53.141691 17709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:53.141845 17709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:53.143030 17709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "d2c4de5fabcb4350a5b65853475472b3"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "47caad08a5262d443440215bedcc6590"
server_key_iv: "5bcbafcc2e5114adc88e2ba4ef17e8dd"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.143422 17709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d2c4de5fabcb4350a5b65853475472b3"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "47caad08a5262d443440215bedcc6590"
server_key_iv: "5bcbafcc2e5114adc88e2ba4ef17e8dd"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.144945 17709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:21:53.145888 17724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.146135 17709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:21:53.146263 17709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "d2c4de5fabcb4350a5b65853475472b3"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "47caad08a5262d443440215bedcc6590"
server_key_iv: "5bcbafcc2e5114adc88e2ba4ef17e8dd"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.146349 17709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:53.161504 17709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:53.163897 17709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:53.164103 17709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:53.164364 17709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:53.164723 17709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:53.164783 17709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.164827 17709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:53.164852 17709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.170960 17709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43647
I20260927 11:21:53.171016 17837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43647 every 8 connection(s)
I20260927 11:21:53.171343 17709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 11:21:53 dist-test-slave-lsvg krb5kdc[17334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:21:53.180341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 17709
I20260927 11:21:53.180460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:21:53.180658 392 external_mini_cluster.cc:1428] Setting key 6de087228f0c076e1e6a0b71c7e64fba
I20260927 11:21:53.181172 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.172092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47223 (local address 127.0.98.62:34443)
0927 11:21:53.172163 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.172165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.172367 (+ 202us) server_negotiation.cc:407] Connection header received
0927 11:21:53.173420 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.173421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.173433 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.173514 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:53.174147 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.174662 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.175650 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.175838 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.175892 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.178167 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:53.178169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:53.178169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:53.178176 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:53.179651 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:53.180067 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:53.180068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:53.180068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:53.180118 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:53.180402 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:53.180403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:53.180403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:53.180592 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:53.180667 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.180972 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.181079 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 11:21:53.181398 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.172049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34443 (local address 127.0.98.3:47223)
0927 11:21:53.172227 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.172273 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:53.173320 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:53.173578 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:53.173582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:53.173843 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:53.174060 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.174065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.174752 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.174752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.175574 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.175576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.175912 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.175913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.176031 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.176520 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:53.176529 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:53.178060 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:53.179741 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:53.179747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:53.179749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:53.179996 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:53.180232 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:53.180233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:53.180233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:53.180335 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:53.180655 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:53.180660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:53.180819 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 11:21:53.180937 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 11:21:53.181148 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260927 11:21:53.182008 17838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34443
I20260927 11:21:53.182120 17838 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:53.182298 17838 heartbeater.cc:511] Master 127.0.98.62:34443 requested a full tablet report, sending...
I20260927 11:21:53.182646 17379 ts_manager.cc:194] Registered new tserver with Master: d2c4de5fabcb4350a5b65853475472b3 (127.0.98.3:43647)
I20260927 11:21:53.183033 17379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47223
I20260927 11:21:53.193248 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:21:53.201247 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.194668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.0.98.62:34443)
0927 11:21:53.194716 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.194723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.194816 (+ 93us) server_negotiation.cc:407] Connection header received
0927 11:21:53.194866 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.194867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.194876 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.194933 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:53.195529 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.196022 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.196783 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.196962 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.197016 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.198324 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:53.198325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:53.198325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:53.198333 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:53.199999 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:53.200376 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:53.200377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:53.200377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:53.200436 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:53.200690 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:53.200691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:53.200691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:53.200842 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:53.200922 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.201037 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.201180 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:21:53.203321 17379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:21:53.205150 17379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:21:53.212098 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.207714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43997 (local address 127.0.0.1:43980)
0927 11:21:53.207994 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.208010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:53.208079 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:53.208441 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:53.208444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:53.208446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:53.208533 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.208535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.209902 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.209903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.210937 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.210940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.211793 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.211794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.211869 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.211882 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 11:21:53.211957 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 11:21:53.212010 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":27,"thread_start_us":92,"threads_started":1}
I20260927 11:21:53.212121 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.207980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.0.98.2:43997)
0927 11:21:53.208271 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.208277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.208286 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:21:53.208326 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.208328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.208341 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.208440 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:53.208622 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.209791 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.211030 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.211699 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.211774 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.211833 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.211991 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.212042 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":83,"threads_started":1}
I20260927 11:21:53.212291 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.207951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36686 (local address 127.0.98.3:43647)
0927 11:21:53.208119 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.208124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.208263 (+ 139us) server_negotiation.cc:407] Connection header received
0927 11:21:53.208401 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.208403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.208421 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.208503 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:53.208778 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.209645 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.211278 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.211777 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.211844 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.211911 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.212140 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.212187 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260927 11:21:53.212311 17849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.207698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35625 (local address 127.0.0.1:55450)
0927 11:21:53.207855 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.208121 (+ 266us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:53.208163 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:53.208734 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:53.208735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:53.208737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:53.208773 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.208774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.210022 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.210023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.211171 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.211174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.212083 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.212084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.212155 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.212160 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:21:53.212193 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 11:21:53.212240 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":32,"thread_start_us":46,"threads_started":1}
I20260927 11:21:53.212661 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.207800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.0.98.1:35625)
0927 11:21:53.208573 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.208577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.208586 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:21:53.208624 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.208626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.208640 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.208721 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:53.208834 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.209899 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.211263 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.211985 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.212404 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.212480 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.212527 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.212573 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":107,"threads_started":1}
I20260927 11:21:53.212428 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.207714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43647 (local address 127.0.0.1:36686)
0927 11:21:53.208192 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.208202 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:53.208256 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:53.208596 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:53.208597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:53.208600 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:53.208674 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.208676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.209734 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.209735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.211184 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.211185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.211946 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.211947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.212000 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.212004 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:21:53.212050 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 11:21:53.212085 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":49,"thread_start_us":89,"threads_started":1}
I20260927 11:21:53.213433 17772 tablet_service.cc:1467] Processing CreateTablet for tablet e025be484d6043a0b18ada6de4d95ee8 (DEFAULT_TABLE table=test-table [id=fe2ec9b5d37041dba00d2fa2ea5ecd94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:53.213439 17638 tablet_service.cc:1467] Processing CreateTablet for tablet e025be484d6043a0b18ada6de4d95ee8 (DEFAULT_TABLE table=test-table [id=fe2ec9b5d37041dba00d2fa2ea5ecd94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:53.213768 17772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e025be484d6043a0b18ada6de4d95ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:53.213776 17638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e025be484d6043a0b18ada6de4d95ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:53.216348 17855 tablet_bootstrap.cc:492] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3: Bootstrap starting.
I20260927 11:21:53.216431 17856 tablet_bootstrap.cc:492] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3: Bootstrap starting.
I20260927 11:21:53.216836 17504 tablet_service.cc:1467] Processing CreateTablet for tablet e025be484d6043a0b18ada6de4d95ee8 (DEFAULT_TABLE table=test-table [id=fe2ec9b5d37041dba00d2fa2ea5ecd94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:53.217036 17856 tablet_bootstrap.cc:654] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:53.217199 17504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e025be484d6043a0b18ada6de4d95ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:53.217264 17855 tablet_bootstrap.cc:654] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:53.217604 17856 log.cc:824] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:53.217792 17855 log.cc:824] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:53.218321 17856 tablet_bootstrap.cc:492] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3: No bootstrap required, opened a new log
I20260927 11:21:53.218413 17856 ts_tablet_manager.cc:1397] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:21:53.219910 17856 raft_consensus.cc:348] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c4de5fabcb4350a5b65853475472b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43647 } } peers { permanent_uuid: "f610bfe38d064de99db3b8776a792a88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35625 } } peers { permanent_uuid: "041083b65faa4f7184e69bf5dd082de3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43997 } }
I20260927 11:21:53.220070 17856 raft_consensus.cc:374] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:53.220059 17859 tablet_bootstrap.cc:492] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88: Bootstrap starting.
I20260927 11:21:53.220118 17856 raft_consensus.cc:729] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041083b65faa4f7184e69bf5dd082de3, State: Initialized, Role: FOLLOWER
I20260927 11:21:53.220234 17856 consensus_queue.cc:260] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c4de5fabcb4350a5b65853475472b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43647 } } peers { permanent_uuid: "f610bfe38d064de99db3b8776a792a88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35625 } } peers { permanent_uuid: "041083b65faa4f7184e69bf5dd082de3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43997 } }
I20260927 11:21:53.220522 17856 ts_tablet_manager.cc:1428] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:21:53.220650 17704 heartbeater.cc:503] Master 127.0.98.62:34443 was elected leader, sending a full tablet report...
I20260927 11:21:53.220803 17859 tablet_bootstrap.cc:654] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:53.221252 17855 tablet_bootstrap.cc:492] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3: No bootstrap required, opened a new log
I20260927 11:21:53.221330 17855 ts_tablet_manager.cc:1397] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 11:21:53.221581 17859 log.cc:824] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:53.222379 17859 tablet_bootstrap.cc:492] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88: No bootstrap required, opened a new log
I20260927 11:21:53.222433 17859 ts_tablet_manager.cc:1397] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 11:21:53.223131 17855 raft_consensus.cc:348] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c4de5fabcb4350a5b65853475472b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43647 } } peers { permanent_uuid: "f610bfe38d064de99db3b8776a792a88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35625 } } peers { permanent_uuid: "041083b65faa4f7184e69bf5dd082de3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43997 } }
I20260927 11:21:53.223302 17855 raft_consensus.cc:374] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:53.223348 17855 raft_consensus.cc:729] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c4de5fabcb4350a5b65853475472b3, State: Initialized, Role: FOLLOWER
I20260927 11:21:53.223474 17855 consensus_queue.cc:260] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c4de5fabcb4350a5b65853475472b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43647 } } peers { permanent_uuid: "f610bfe38d064de99db3b8776a792a88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35625 } } peers { permanent_uuid: "041083b65faa4f7184e69bf5dd082de3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43997 } }
I20260927 11:21:53.223752 17855 ts_tablet_manager.cc:1428] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:21:53.223789 17838 heartbeater.cc:503] Master 127.0.98.62:34443 was elected leader, sending a full tablet report...
I20260927 11:21:53.224100 17859 raft_consensus.cc:348] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c4de5fabcb4350a5b65853475472b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43647 } } peers { permanent_uuid: "f610bfe38d064de99db3b8776a792a88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35625 } } peers { permanent_uuid: "041083b65faa4f7184e69bf5dd082de3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43997 } }
I20260927 11:21:53.224280 17859 raft_consensus.cc:374] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:53.224319 17859 raft_consensus.cc:729] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f610bfe38d064de99db3b8776a792a88, State: Initialized, Role: FOLLOWER
I20260927 11:21:53.224481 17859 consensus_queue.cc:260] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c4de5fabcb4350a5b65853475472b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43647 } } peers { permanent_uuid: "f610bfe38d064de99db3b8776a792a88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35625 } } peers { permanent_uuid: "041083b65faa4f7184e69bf5dd082de3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43997 } }
I20260927 11:21:53.224807 17859 ts_tablet_manager.cc:1428] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 11:21:53.224821 17570 heartbeater.cc:503] Master 127.0.98.62:34443 was elected leader, sending a full tablet report...
I20260927 11:21:53.236586 17860 raft_consensus.cc:482] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:21:53.236698 17860 raft_consensus.cc:504] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c4de5fabcb4350a5b65853475472b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43647 } } peers { permanent_uuid: "f610bfe38d064de99db3b8776a792a88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35625 } } peers { permanent_uuid: "041083b65faa4f7184e69bf5dd082de3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43997 } }
I20260927 11:21:53.237178 17860 leader_election.cc:288] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2c4de5fabcb4350a5b65853475472b3 (127.0.98.3:43647), f610bfe38d064de99db3b8776a792a88 (127.0.98.1:35625)
I20260927 11:21:53.240021 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.237384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53483 (local address 127.0.98.3:43647)
0927 11:21:53.237439 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.237441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.237450 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:21:53.237529 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.237532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.237550 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.237626 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:53.237792 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.238268 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.239223 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.239662 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.239746 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.239757 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.239908 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.239945 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 11:21:53.240068 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.237313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43647 (local address 127.0.98.2:53483)
0927 11:21:53.237382 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.237394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:53.237455 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:53.237633 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:53.237634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:53.237636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:53.237710 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.237712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.238365 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.238367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.239127 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.239129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.239744 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.239745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.239823 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.239826 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:21:53.239902 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 11:21:53.239943 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 11:21:53.240322 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.237320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35625 (local address 127.0.98.2:41325)
0927 11:21:53.237611 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.237621 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:53.237685 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:53.237807 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:53.237808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:53.237809 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:53.237887 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.237888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.238511 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.238512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.239365 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.239368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.239948 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.239950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.240055 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.240058 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:21:53.240211 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 11:21:53.240250 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":105,"thread_start_us":135,"threads_started":1}
I20260927 11:21:53.240374 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.237404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41325 (local address 127.0.98.1:35625)
0927 11:21:53.237452 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.237455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.237671 (+ 216us) server_negotiation.cc:407] Connection header received
0927 11:21:53.237736 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.237738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.237750 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.237804 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:53.237964 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.238428 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.239439 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.239861 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.239936 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.239946 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.240266 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.240311 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:21:53.240410 17792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e025be484d6043a0b18ada6de4d95ee8" candidate_uuid: "041083b65faa4f7184e69bf5dd082de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c4de5fabcb4350a5b65853475472b3" is_pre_election: true
I20260927 11:21:53.240572 17792 raft_consensus.cc:2454] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 041083b65faa4f7184e69bf5dd082de3 in term 0.
I20260927 11:21:53.240748 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e025be484d6043a0b18ada6de4d95ee8" candidate_uuid: "041083b65faa4f7184e69bf5dd082de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f610bfe38d064de99db3b8776a792a88" is_pre_election: true
I20260927 11:21:53.240808 17591 leader_election.cc:302] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041083b65faa4f7184e69bf5dd082de3, d2c4de5fabcb4350a5b65853475472b3; no voters:
I20260927 11:21:53.240929 17524 raft_consensus.cc:2454] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 041083b65faa4f7184e69bf5dd082de3 in term 0.
I20260927 11:21:53.240931 17860 raft_consensus.cc:2781] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:21:53.240978 17860 raft_consensus.cc:482] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:21:53.241012 17860 raft_consensus.cc:3037] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:53.241707 17860 raft_consensus.cc:504] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c4de5fabcb4350a5b65853475472b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43647 } } peers { permanent_uuid: "f610bfe38d064de99db3b8776a792a88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35625 } } peers { permanent_uuid: "041083b65faa4f7184e69bf5dd082de3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43997 } }
I20260927 11:21:53.241856 17860 leader_election.cc:288] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [CANDIDATE]: Term 1 election: Requested vote from peers d2c4de5fabcb4350a5b65853475472b3 (127.0.98.3:43647), f610bfe38d064de99db3b8776a792a88 (127.0.98.1:35625)
I20260927 11:21:53.241971 17792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e025be484d6043a0b18ada6de4d95ee8" candidate_uuid: "041083b65faa4f7184e69bf5dd082de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c4de5fabcb4350a5b65853475472b3"
I20260927 11:21:53.242048 17792 raft_consensus.cc:3037] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:53.242056 17524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e025be484d6043a0b18ada6de4d95ee8" candidate_uuid: "041083b65faa4f7184e69bf5dd082de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f610bfe38d064de99db3b8776a792a88"
I20260927 11:21:53.242192 17524 raft_consensus.cc:3037] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:53.242811 17792 raft_consensus.cc:2454] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 041083b65faa4f7184e69bf5dd082de3 in term 1.
I20260927 11:21:53.243026 17524 raft_consensus.cc:2454] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 041083b65faa4f7184e69bf5dd082de3 in term 1.
I20260927 11:21:53.243075 17591 leader_election.cc:302] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 041083b65faa4f7184e69bf5dd082de3, d2c4de5fabcb4350a5b65853475472b3; no voters:
I20260927 11:21:53.243198 17860 raft_consensus.cc:2781] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:21:53.243352 17860 raft_consensus.cc:686] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [term 1 LEADER]: Becoming Leader. State: Replica: 041083b65faa4f7184e69bf5dd082de3, State: Running, Role: LEADER
I20260927 11:21:53.243449 17860 consensus_queue.cc:237] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c4de5fabcb4350a5b65853475472b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43647 } } peers { permanent_uuid: "f610bfe38d064de99db3b8776a792a88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35625 } } peers { permanent_uuid: "041083b65faa4f7184e69bf5dd082de3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43997 } }
I20260927 11:21:53.244436 17378 catalog_manager.cc:5495] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 041083b65faa4f7184e69bf5dd082de3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "041083b65faa4f7184e69bf5dd082de3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2c4de5fabcb4350a5b65853475472b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f610bfe38d064de99db3b8776a792a88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "041083b65faa4f7184e69bf5dd082de3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43997 } health_report { overall_health: HEALTHY } } }
I20260927 11:21:53.259140 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.256729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.0.98.2:43997)
0927 11:21:53.256794 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.256796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.256806 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:21:53.256890 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.256891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.256910 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.256974 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:21:53.257179 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.257641 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.258403 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.258597 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.258654 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.258779 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:21:53.258877 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:21:53.258940 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.258988 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.259006 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 11:21:53.261251 17792 raft_consensus.cc:1261] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3 [term 1 FOLLOWER]: Refusing update from remote peer 041083b65faa4f7184e69bf5dd082de3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:53.261255 17524 raft_consensus.cc:1261] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88 [term 1 FOLLOWER]: Refusing update from remote peer 041083b65faa4f7184e69bf5dd082de3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:53.261693 17865 consensus_queue.cc:1035] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f610bfe38d064de99db3b8776a792a88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:53.261862 17860 consensus_queue.cc:1035] T e025be484d6043a0b18ada6de4d95ee8 P 041083b65faa4f7184e69bf5dd082de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2c4de5fabcb4350a5b65853475472b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:53.264928 17868 mvcc.cc:205] Tried to move back new op lower bound from 7333921231915089920 to 7333921231846162432. Current Snapshot: MvccSnapshot[applied={T|T < 7333921231915089920}]
I20260927 11:21:53.265662 17871 mvcc.cc:205] Tried to move back new op lower bound from 7333921231915089920 to 7333921231846162432. Current Snapshot: MvccSnapshot[applied={T|T < 7333921231915089920}]
I20260927 11:21:53.267705 17379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:53.267832 17379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:53.268651 17379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2605df2874364c968930ce7c42ebd7ec: Sending DeleteTablet for 3 replicas of tablet e025be484d6043a0b18ada6de4d95ee8
I20260927 11:21:53.268988 17772 tablet_service.cc:1514] Processing DeleteTablet for tablet e025be484d6043a0b18ada6de4d95ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36686
I20260927 11:21:53.269006 17638 tablet_service.cc:1514] Processing DeleteTablet for tablet e025be484d6043a0b18ada6de4d95ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43980
I20260927 11:21:53.269265 17877 tablet_replica.cc:330] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3: stopping tablet replica
I20260927 11:21:53.269258 17504 tablet_service.cc:1514] Processing DeleteTablet for tablet e025be484d6043a0b18ada6de4d95ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55450
I20260927 11:21:53.269358 17877 raft_consensus.cc:2229] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:21:53.269471 17877 raft_consensus.cc:2258] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:53.269480 17879 tablet_replica.cc:330] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88: stopping tablet replica
I20260927 11:21:53.269572 17879 raft_consensus.cc:2229] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:21:53.269688 17879 raft_consensus.cc:2258] T e025be484d6043a0b18ada6de4d95ee8 P f610bfe38d064de99db3b8776a792a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:53.269996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 17441
I20260927 11:21:53.270066 17877 ts_tablet_manager.cc:1905] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:53.272383 17877 ts_tablet_manager.cc:1918] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:53.272456 17877 log.cc:1196] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/e025be484d6043a0b18ada6de4d95ee8
I20260927 11:21:53.272737 17877 ts_tablet_manager.cc:1939] T e025be484d6043a0b18ada6de4d95ee8 P d2c4de5fabcb4350a5b65853475472b3: Deleting consensus metadata
I20260927 11:21:53.273197 17363 catalog_manager.cc:4841] TS d2c4de5fabcb4350a5b65853475472b3 (127.0.98.3:43647): tablet e025be484d6043a0b18ada6de4d95ee8 (table test-table [id=fe2ec9b5d37041dba00d2fa2ea5ecd94]) successfully deleted
W20260927 11:21:53.275986 17365 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35625 (error 108)
I20260927 11:21:53.276093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 17575
I20260927 11:21:53.276952 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.276827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35625 (local address 127.0.0.1:55458)
0927 11:21:53.276877 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.276913 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 11:21:53.277101 17365 catalog_manager.cc:4584] TS f610bfe38d064de99db3b8776a792a88 (127.0.98.1:35625): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e025be484d6043a0b18ada6de4d95ee8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35625: connect: Connection refused (error 111)
I20260927 11:21:53.281136 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 17709
I20260927 11:21:53.281199 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.281094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43997 (local address 127.0.0.1:43994)
0927 11:21:53.281132 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.281163 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 11:21:53.285957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 17347
2026-09-27T11:21:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1958 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T11:21:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:21:53Z Disabled control of system clock
I20260927 11:21:53.314352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37099
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:21:53.397377 17884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:53.397611 17884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:53.397634 17884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:53.398764 17884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:21:53.398801 17884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:53.398814 17884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:53.398828 17884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:21:53.398839 17884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:21:53.400113 17884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:53.400310 17884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:53.400555 17884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:53.405552 17889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:53.405570 17892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:53.405764 17884 server_base.cc:1034] running on GCE node
W20260927 11:21:53.405571 17890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:53.406073 17884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:53.406325 17884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:53.407495 17884 hybrid_clock.cc:648] HybridClock initialized: now 1790508113407479 us; error 41 us; skew 500 ppm
I20260927 11:21:53.408800 17884 webserver.cc:460] Webserver started at http://127.0.98.62:37683/ using document root <none> and password file <none>
I20260927 11:21:53.409086 17884 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:53.409153 17884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:53.409296 17884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:53.410422 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "1fc3405fea9343c99d794d922d1a18ca"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "a39b11ac77bdd8130e3a6c5ef6dc6536"
server_key_iv: "65890d547014654b02d672bedd69d935"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.410789 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "1fc3405fea9343c99d794d922d1a18ca"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "a39b11ac77bdd8130e3a6c5ef6dc6536"
server_key_iv: "65890d547014654b02d672bedd69d935"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.412107 17884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:21:53.412812 17898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.412984 17884 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:21:53.413101 17884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "1fc3405fea9343c99d794d922d1a18ca"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "a39b11ac77bdd8130e3a6c5ef6dc6536"
server_key_iv: "65890d547014654b02d672bedd69d935"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.413190 17884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:53.444238 17884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:53.444625 17884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:53.444806 17884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:53.449239 17884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37099
I20260927 11:21:53.449275 17950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37099 every 8 connection(s)
I20260927 11:21:53.449661 17884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:21:53.450649 17951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:53.452943 17951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca: Bootstrap starting.
I20260927 11:21:53.453574 17951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:53.453913 17951 log.cc:824] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:53.454641 17951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca: No bootstrap required, opened a new log
I20260927 11:21:53.456316 17951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc3405fea9343c99d794d922d1a18ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37099 } }
I20260927 11:21:53.456455 17951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:53.456480 17951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc3405fea9343c99d794d922d1a18ca, State: Initialized, Role: FOLLOWER
I20260927 11:21:53.456660 17951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc3405fea9343c99d794d922d1a18ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37099 } }
I20260927 11:21:53.456737 17951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:21:53.456760 17951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:21:53.456828 17951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:53.457446 17951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc3405fea9343c99d794d922d1a18ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37099 } }
I20260927 11:21:53.457533 17951 leader_election.cc:302] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fc3405fea9343c99d794d922d1a18ca; no voters:
I20260927 11:21:53.457717 17951 leader_election.cc:288] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:21:53.457789 17954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:21:53.457975 17954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [term 1 LEADER]: Becoming Leader. State: Replica: 1fc3405fea9343c99d794d922d1a18ca, State: Running, Role: LEADER
I20260927 11:21:53.457993 17951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:21:53.458148 17954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc3405fea9343c99d794d922d1a18ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37099 } }
I20260927 11:21:53.458575 17956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fc3405fea9343c99d794d922d1a18ca. Latest consensus state: current_term: 1 leader_uuid: "1fc3405fea9343c99d794d922d1a18ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc3405fea9343c99d794d922d1a18ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37099 } } }
I20260927 11:21:53.458669 17955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fc3405fea9343c99d794d922d1a18ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fc3405fea9343c99d794d922d1a18ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37099 } } }
I20260927 11:21:53.459014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 17884
I20260927 11:21:53.459117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:21:53.459347 392 external_mini_cluster.cc:1428] Setting key 89b13b865d97f23924104674dcf64f1c
I20260927 11:21:53.458724 17956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [sys.catalog]: This master's current role is: LEADER
I20260927 11:21:53.459738 17955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca [sys.catalog]: This master's current role is: LEADER
W20260927 11:21:53.460006 17969 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:21:53.460162 17969 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:21:53.460290 17970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:21:53.460469 17970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:21:53.462069 17970 catalog_manager.cc:1347] Generated new cluster ID: 9940e378db8149288a84b71062903fd4
I20260927 11:21:53.462122 17970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:21:53.465750 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.459842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59150 (local address 127.0.98.62:37099)
0927 11:21:53.460103 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.460113 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.460303 (+ 190us) server_negotiation.cc:407] Connection header received
0927 11:21:53.460440 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.460448 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.460527 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.460690 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:53.461075 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.462308 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.463385 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.463976 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.464075 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.464944 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:53.464948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:53.464948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:53.465187 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:53.465227 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:53.465241 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:53.465292 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.465473 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.465547 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":64,"threads_started":1}
I20260927 11:21:53.474958 17970 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:21:53.475476 17970 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:21:53.479394 17970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca: Generated new TSK 0
I20260927 11:21:53.479579 17970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:21:53.482164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37099
--builtin_ntp_servers=127.0.98.20:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:21:53.563436 17975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:53.563694 17975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:53.563735 17975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:53.564921 17975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:53.564985 17975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:53.565029 17975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:21:53.566454 17975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:53.566740 17975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:53.567049 17975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:53.572854 17980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:53.572866 17981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:53.572971 17983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:53.573227 17975 server_base.cc:1034] running on GCE node
I20260927 11:21:53.573455 17975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:53.573702 17975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:53.574875 17975 hybrid_clock.cc:648] HybridClock initialized: now 1790508113574857 us; error 34 us; skew 500 ppm
I20260927 11:21:53.576134 17975 webserver.cc:460] Webserver started at http://127.0.98.1:34729/ using document root <none> and password file <none>
I20260927 11:21:53.576396 17975 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:53.576470 17975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:53.576625 17975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:53.577761 17975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "816ec746c35042278bbb9794c9c4f38f"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "5e03ce68f04cf94dc13640a2b6ce0b3f"
server_key_iv: "57403a0717c781bba19d1b8cc4604aa3"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.578143 17975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "816ec746c35042278bbb9794c9c4f38f"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "5e03ce68f04cf94dc13640a2b6ce0b3f"
server_key_iv: "57403a0717c781bba19d1b8cc4604aa3"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.579464 17975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:21:53.580279 17989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.580480 17975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:53.580576 17975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "816ec746c35042278bbb9794c9c4f38f"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "5e03ce68f04cf94dc13640a2b6ce0b3f"
server_key_iv: "57403a0717c781bba19d1b8cc4604aa3"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.580664 17975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:53.602174 17975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:53.602521 17975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:53.602695 17975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:53.602944 17975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:53.603295 17975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:53.603349 17975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.603391 17975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:53.603420 17975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.609316 17975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33747
I20260927 11:21:53.609375 18102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33747 every 8 connection(s)
I20260927 11:21:53.609741 17975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:21:53.615083 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.610553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42363 (local address 127.0.98.62:37099)
0927 11:21:53.610631 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.610634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.610879 (+ 245us) server_negotiation.cc:407] Connection header received
0927 11:21:53.611346 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.611347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.611361 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.611418 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:53.612090 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.612574 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.613596 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.613783 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.613837 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.614831 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:53.614832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:53.614833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:53.614837 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:53.614851 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:53.614855 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:53.614898 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.614994 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.615024 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 11:21:53.615284 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.610520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37099 (local address 127.0.98.1:42363)
0927 11:21:53.610751 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.610786 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:53.611265 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:53.611510 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:53.611514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:53.611522 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:53.612017 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.612021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.612651 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.612652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.613524 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.613527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.613878 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.613879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.613996 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.614691 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:21:53.614692 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:53.614721 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:21:53.614743 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:53.614925 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:53.614929 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:53.614931 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:21:53.614985 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 11:21:53.615066 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":113,"threads_started":1}
I20260927 11:21:53.615794 18103 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37099
I20260927 11:21:53.615898 18103 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:53.616051 18103 heartbeater.cc:511] Master 127.0.98.62:37099 requested a full tablet report, sending...
I20260927 11:21:53.616640 17915 ts_manager.cc:194] Registered new tserver with Master: 816ec746c35042278bbb9794c9c4f38f (127.0.98.1:33747)
I20260927 11:21:53.617303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 17975
I20260927 11:21:53.617312 17915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42363
I20260927 11:21:53.617465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:21:53.617725 392 external_mini_cluster.cc:1428] Setting key 7429e442da66d367eb1c6a889ce42115
I20260927 11:21:53.618757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37099
--builtin_ntp_servers=127.0.98.20:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:21:53.699759 18106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:53.699988 18106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:53.700026 18106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:53.701241 18106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:53.701299 18106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:53.701344 18106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:21:53.702698 18106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:53.702958 18106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:53.703253 18106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:53.708194 18111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:53.708194 18112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:53.708267 18114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:53.708350 18106 server_base.cc:1034] running on GCE node
I20260927 11:21:53.708671 18106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:53.708873 18106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:53.710006 18106 hybrid_clock.cc:648] HybridClock initialized: now 1790508113709987 us; error 48 us; skew 500 ppm
I20260927 11:21:53.711308 18106 webserver.cc:460] Webserver started at http://127.0.98.2:46813/ using document root <none> and password file <none>
I20260927 11:21:53.711563 18106 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:53.711622 18106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:53.711799 18106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:53.712845 18106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "63eff2629e9c49d09f26fff405c0b2f3"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "ba4b65ba47757a096aefa12fded645c2"
server_key_iv: "7d1274edde660aa367a0a3150ccde759"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.713279 18106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "63eff2629e9c49d09f26fff405c0b2f3"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "ba4b65ba47757a096aefa12fded645c2"
server_key_iv: "7d1274edde660aa367a0a3150ccde759"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.714568 18106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:53.715304 18120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.715551 18106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:53.715646 18106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "63eff2629e9c49d09f26fff405c0b2f3"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "ba4b65ba47757a096aefa12fded645c2"
server_key_iv: "7d1274edde660aa367a0a3150ccde759"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.715718 18106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:53.734964 18106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:53.735319 18106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:53.735489 18106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:53.735741 18106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:53.736099 18106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:53.736161 18106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.736203 18106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:53.736249 18106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.742029 18106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35221
I20260927 11:21:53.742071 18233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35221 every 8 connection(s)
I20260927 11:21:53.742446 18106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:21:53.743584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 18106
I20260927 11:21:53.743693 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:21:53.743907 392 external_mini_cluster.cc:1428] Setting key 90614f906d5f502340c58b05f4fc6fe8
I20260927 11:21:53.745033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37099
--builtin_ntp_servers=127.0.98.20:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 11:21:53.748409 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.743281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47287 (local address 127.0.98.62:37099)
0927 11:21:53.743375 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.743377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.743566 (+ 189us) server_negotiation.cc:407] Connection header received
0927 11:21:53.744326 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.744327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.744339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.744405 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:53.745329 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.745808 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.746991 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.747198 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.747289 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.748155 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:53.748156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:53.748156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:53.748160 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:53.748174 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:53.748178 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:53.748222 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.748327 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.748350 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 11:21:53.748697 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.743197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37099 (local address 127.0.98.2:47287)
0927 11:21:53.743406 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.743455 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:53.744209 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:53.744461 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:53.744464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:53.744473 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:53.745251 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.745258 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.745909 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.745910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.746895 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.746897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.747300 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.747301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.747461 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.748002 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:21:53.748003 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:53.748041 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:21:53.748069 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:53.748242 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:53.748247 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:53.748250 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:21:53.748319 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 11:21:53.748430 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260927 11:21:53.749219 18234 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37099
I20260927 11:21:53.749313 18234 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:53.749511 18234 heartbeater.cc:511] Master 127.0.98.62:37099 requested a full tablet report, sending...
I20260927 11:21:53.749889 17915 ts_manager.cc:194] Registered new tserver with Master: 63eff2629e9c49d09f26fff405c0b2f3 (127.0.98.2:35221)
I20260927 11:21:53.750259 17915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47287
W20260927 11:21:53.826687 18237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:53.826938 18237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:53.826980 18237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:53.828154 18237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:53.828226 18237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:53.828269 18237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:21:53.829707 18237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:53.829977 18237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:53.830282 18237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:53.835399 18243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:53.835498 18237 server_base.cc:1034] running on GCE node
W20260927 11:21:53.835476 18245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:53.835404 18242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:53.835896 18237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:53.836122 18237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:53.837270 18237 hybrid_clock.cc:648] HybridClock initialized: now 1790508113837253 us; error 43 us; skew 500 ppm
I20260927 11:21:53.838613 18237 webserver.cc:460] Webserver started at http://127.0.98.3:45119/ using document root <none> and password file <none>
I20260927 11:21:53.838836 18237 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:53.838888 18237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:53.838989 18237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:53.840063 18237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "dfcfb5da82334208bcc3e5a82db4ef76"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "ed76d0022529bbf5340ce3f14f7109d0"
server_key_iv: "7060e84147c26e4d2ea0fdf02f8d833c"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.840391 18237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dfcfb5da82334208bcc3e5a82db4ef76"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "ed76d0022529bbf5340ce3f14f7109d0"
server_key_iv: "7060e84147c26e4d2ea0fdf02f8d833c"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.841722 18237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:21:53.842403 18251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.842576 18237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:21:53.842679 18237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "dfcfb5da82334208bcc3e5a82db4ef76"
format_stamp: "Formatted at 2026-09-27 11:21:53 on dist-test-slave-lsvg"
server_key: "ed76d0022529bbf5340ce3f14f7109d0"
server_key_iv: "7060e84147c26e4d2ea0fdf02f8d833c"
server_key_version: "encryptionkey@0"
I20260927 11:21:53.842756 18237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:53.858642 18237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:53.858997 18237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:53.859177 18237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:53.859436 18237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:53.859779 18237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:53.859834 18237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.859874 18237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:53.859903 18237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:53.866063 18237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36825
I20260927 11:21:53.866107 18364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36825 every 8 connection(s)
I20260927 11:21:53.866511 18237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:21:53.870345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 18237
I20260927 11:21:53.870481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:21:53.870661 392 external_mini_cluster.cc:1428] Setting key c75cfa280f0391df1e26c9db655b23fa
I20260927 11:21:53.872378 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.867439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57175 (local address 127.0.98.62:37099)
0927 11:21:53.867522 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.867524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.867793 (+ 269us) server_negotiation.cc:407] Connection header received
0927 11:21:53.868266 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.868266 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.868277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.868336 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:53.869242 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.869736 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.870861 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.871159 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.871228 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.872150 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:53.872151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:53.872151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:53.872155 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:53.872168 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:53.872172 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:53.872217 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.872296 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.872321 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 11:21:53.872610 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.867443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37099 (local address 127.0.98.3:57175)
0927 11:21:53.867645 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.867680 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:53.868181 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:53.868413 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:53.868416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:53.868424 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:53.868984 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.868989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.869829 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.869830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.870751 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.870754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.871266 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.871268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.871429 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.872020 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:21:53.872021 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:53.872051 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:21:53.872073 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:53.872229 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:53.872232 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:53.872234 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:21:53.872287 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 11:21:53.872368 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260927 11:21:53.873168 18365 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37099
I20260927 11:21:53.873267 18365 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:53.873443 18365 heartbeater.cc:511] Master 127.0.98.62:37099 requested a full tablet report, sending...
I20260927 11:21:53.873799 17915 ts_manager.cc:194] Registered new tserver with Master: dfcfb5da82334208bcc3e5a82db4ef76 (127.0.98.3:36825)
I20260927 11:21:53.874130 17915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57175
I20260927 11:21:53.882738 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:21:53.887214 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.884218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.0.98.62:37099)
0927 11:21:53.884300 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.884302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.884370 (+ 68us) server_negotiation.cc:407] Connection header received
0927 11:21:53.884460 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.884461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.884473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.884530 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:53.884730 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.885289 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.886034 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.886234 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.886294 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.886963 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:53.886964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:53.886964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:53.886968 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:53.886981 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:53.886985 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:53.887031 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.887079 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.887102 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 11:21:53.888945 17915 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:21:53.890580 17915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:21:53.896646 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.893073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33747 (local address 127.0.0.1:42836)
0927 11:21:53.893200 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.893230 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:53.893293 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:53.893675 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:53.893676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:53.893677 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:53.893738 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.893739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.894592 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.894594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.895648 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.895650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.896337 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.896338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.896418 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.896430 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 11:21:53.896514 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 11:21:53.896565 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260927 11:21:53.896657 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.893163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.0.98.1:33747)
0927 11:21:53.893352 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.893358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.893369 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:21:53.893515 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.893518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.893535 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.893605 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:53.893815 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.894505 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.895740 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.896237 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.896333 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.896440 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.896528 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.896568 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260927 11:21:53.897563 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.892943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.0.98.2:35221)
0927 11:21:53.893451 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.893456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.893466 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:21:53.893503 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.893505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.893522 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.893590 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:53.893832 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.894951 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.896420 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.897180 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.897255 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.897327 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.897411 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.897451 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":92,"threads_started":1}
I20260927 11:21:53.897683 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.892896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.0.98.3:36825)
0927 11:21:53.893122 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:53.893128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:53.893288 (+ 160us) server_negotiation.cc:407] Connection header received
0927 11:21:53.893472 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:53.893475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:53.893493 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:53.893552 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:53.893838 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.894948 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.896433 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.897180 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.897230 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.897311 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:53.897544 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 11:21:53.897592 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20260927 11:21:53.897795 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.892846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35221 (local address 127.0.0.1:56758)
0927 11:21:53.893197 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.893230 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:53.893293 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:53.893663 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:53.893666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:53.893670 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:53.893752 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.893753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.895040 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.895041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.896312 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.896315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.897264 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.897265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.897328 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.897333 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:21:53.897396 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 11:21:53.897432 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":147,"thread_start_us":113,"threads_started":1}
I20260927 11:21:53.897732 18037 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7e7a1abe104ca2b3193b204f07e3e1 (DEFAULT_TABLE table=test-table [id=96fef1b4158a45ec8fa8e5da59ae7886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:53.897943 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:53.892810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36825 (local address 127.0.0.1:51682)
0927 11:21:53.893169 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:53.893215 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:53.893325 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:53.893663 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:53.893666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:53.893668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:53.893751 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.893753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.895051 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.895052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.896312 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:53.896314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:53.897374 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:53.897375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:53.897448 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:53.897451 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:21:53.897827 (+ 376us) client_negotiation.cc:240] Negotiation successful
0927 11:21:53.897869 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":33,"thread_start_us":89,"threads_started":1}
I20260927 11:21:53.898167 18037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7e7a1abe104ca2b3193b204f07e3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:53.898620 18299 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7e7a1abe104ca2b3193b204f07e3e1 (DEFAULT_TABLE table=test-table [id=96fef1b4158a45ec8fa8e5da59ae7886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:53.898617 18168 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7e7a1abe104ca2b3193b204f07e3e1 (DEFAULT_TABLE table=test-table [id=96fef1b4158a45ec8fa8e5da59ae7886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:53.898933 18299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7e7a1abe104ca2b3193b204f07e3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:53.898944 18168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7e7a1abe104ca2b3193b204f07e3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:53.900805 18379 tablet_bootstrap.cc:492] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f: Bootstrap starting.
I20260927 11:21:53.901484 18379 tablet_bootstrap.cc:654] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:53.901790 18379 log.cc:824] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:53.902130 18381 tablet_bootstrap.cc:492] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3: Bootstrap starting.
I20260927 11:21:53.902689 18381 tablet_bootstrap.cc:654] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:53.902918 18381 log.cc:824] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:53.903178 18380 tablet_bootstrap.cc:492] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76: Bootstrap starting.
I20260927 11:21:53.903631 18381 tablet_bootstrap.cc:492] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3: No bootstrap required, opened a new log
I20260927 11:21:53.903728 18381 ts_tablet_manager.cc:1397] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:21:53.903741 18380 tablet_bootstrap.cc:654] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:53.904069 18380 log.cc:824] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:53.904878 18380 tablet_bootstrap.cc:492] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76: No bootstrap required, opened a new log
I20260927 11:21:53.904958 18380 ts_tablet_manager.cc:1397] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:21:53.905361 18379 tablet_bootstrap.cc:492] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f: No bootstrap required, opened a new log
I20260927 11:21:53.905448 18379 ts_tablet_manager.cc:1397] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260927 11:21:53.906559 18381 raft_consensus.cc:348] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcfb5da82334208bcc3e5a82db4ef76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36825 } } peers { permanent_uuid: "63eff2629e9c49d09f26fff405c0b2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35221 } } peers { permanent_uuid: "816ec746c35042278bbb9794c9c4f38f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33747 } }
I20260927 11:21:53.906800 18381 raft_consensus.cc:374] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:53.906898 18381 raft_consensus.cc:729] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63eff2629e9c49d09f26fff405c0b2f3, State: Initialized, Role: FOLLOWER
I20260927 11:21:53.907071 18381 consensus_queue.cc:260] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcfb5da82334208bcc3e5a82db4ef76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36825 } } peers { permanent_uuid: "63eff2629e9c49d09f26fff405c0b2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35221 } } peers { permanent_uuid: "816ec746c35042278bbb9794c9c4f38f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33747 } }
I20260927 11:21:53.907444 18381 ts_tablet_manager.cc:1428] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260927 11:21:53.907457 18234 heartbeater.cc:503] Master 127.0.98.62:37099 was elected leader, sending a full tablet report...
I20260927 11:21:53.907553 18379 raft_consensus.cc:348] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcfb5da82334208bcc3e5a82db4ef76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36825 } } peers { permanent_uuid: "63eff2629e9c49d09f26fff405c0b2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35221 } } peers { permanent_uuid: "816ec746c35042278bbb9794c9c4f38f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33747 } }
I20260927 11:21:53.907708 18379 raft_consensus.cc:374] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:53.907740 18379 raft_consensus.cc:729] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816ec746c35042278bbb9794c9c4f38f, State: Initialized, Role: FOLLOWER
I20260927 11:21:53.907716 18380 raft_consensus.cc:348] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcfb5da82334208bcc3e5a82db4ef76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36825 } } peers { permanent_uuid: "63eff2629e9c49d09f26fff405c0b2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35221 } } peers { permanent_uuid: "816ec746c35042278bbb9794c9c4f38f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33747 } }
I20260927 11:21:53.907853 18379 consensus_queue.cc:260] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcfb5da82334208bcc3e5a82db4ef76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36825 } } peers { permanent_uuid: "63eff2629e9c49d09f26fff405c0b2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35221 } } peers { permanent_uuid: "816ec746c35042278bbb9794c9c4f38f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33747 } }
I20260927 11:21:53.908114 18379 ts_tablet_manager.cc:1428] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 11:21:53.907864 18380 raft_consensus.cc:374] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:53.908218 18103 heartbeater.cc:503] Master 127.0.98.62:37099 was elected leader, sending a full tablet report...
I20260927 11:21:53.908203 18380 raft_consensus.cc:729] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcfb5da82334208bcc3e5a82db4ef76, State: Initialized, Role: FOLLOWER
I20260927 11:21:53.908368 18380 consensus_queue.cc:260] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcfb5da82334208bcc3e5a82db4ef76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36825 } } peers { permanent_uuid: "63eff2629e9c49d09f26fff405c0b2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35221 } } peers { permanent_uuid: "816ec746c35042278bbb9794c9c4f38f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33747 } }
I20260927 11:21:53.908659 18380 ts_tablet_manager.cc:1428] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260927 11:21:53.908768 18365 heartbeater.cc:503] Master 127.0.98.62:37099 was elected leader, sending a full tablet report...
W20260927 11:21:53.993026 18235 tablet.cc:2369] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:21:54.108954 18386 raft_consensus.cc:482] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:21:54.109179 18386 raft_consensus.cc:504] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcfb5da82334208bcc3e5a82db4ef76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36825 } } peers { permanent_uuid: "63eff2629e9c49d09f26fff405c0b2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35221 } } peers { permanent_uuid: "816ec746c35042278bbb9794c9c4f38f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33747 } }
I20260927 11:21:54.109517 18386 leader_election.cc:288] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfcfb5da82334208bcc3e5a82db4ef76 (127.0.98.3:36825), 63eff2629e9c49d09f26fff405c0b2f3 (127.0.98.2:35221)
W20260927 11:21:54.110384 18104 tablet.cc:2369] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:21:54.112496 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:54.109828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33333 (local address 127.0.98.2:35221)
0927 11:21:54.109905 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:54.109907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:54.109916 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:21:54.109953 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:54.109954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:54.109961 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:54.110024 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:54.110189 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:54.110831 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:54.111770 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:54.112149 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:54.112232 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:54.112242 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:54.112391 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 11:21:54.112431 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 11:21:54.112496 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:54.109723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35221 (local address 127.0.98.1:33333)
0927 11:21:54.109762 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:54.109774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:54.109821 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:54.110035 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:54.110036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:54.110038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:54.110115 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:54.110117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:54.110925 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:54.110926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:54.111675 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:54.111677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:54.112233 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:54.112234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:54.112306 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:54.112309 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:21:54.112383 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 11:21:54.112425 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 11:21:54.112571 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:54.109825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34505 (local address 127.0.98.3:36825)
0927 11:21:54.109894 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:54.109896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:54.110088 (+ 192us) server_negotiation.cc:407] Connection header received
0927 11:21:54.110122 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:54.110122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:54.110131 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:54.110201 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:54.110353 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:54.110904 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:54.111856 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:54.112213 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:54.112291 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:54.112300 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:54.112467 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 11:21:54.112500 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 11:21:54.112603 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:54.109723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36825 (local address 127.0.98.1:34505)
0927 11:21:54.110005 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:54.110015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:54.110075 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:54.110216 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:54.110216 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:54.110218 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:54.110275 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:54.110276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:54.110993 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:54.110995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:54.111730 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:54.111732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:54.112306 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:54.112307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:54.112391 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:54.112394 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:21:54.112461 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 11:21:54.112499 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":149,"thread_start_us":77,"threads_started":1}
I20260927 11:21:54.112943 18319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7e7a1abe104ca2b3193b204f07e3e1" candidate_uuid: "816ec746c35042278bbb9794c9c4f38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcfb5da82334208bcc3e5a82db4ef76" is_pre_election: true
I20260927 11:21:54.113160 18319 raft_consensus.cc:2454] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816ec746c35042278bbb9794c9c4f38f in term 0.
I20260927 11:21:54.113095 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7e7a1abe104ca2b3193b204f07e3e1" candidate_uuid: "816ec746c35042278bbb9794c9c4f38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63eff2629e9c49d09f26fff405c0b2f3" is_pre_election: true
I20260927 11:21:54.113263 18188 raft_consensus.cc:2454] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 816ec746c35042278bbb9794c9c4f38f in term 0.
I20260927 11:21:54.113432 17993 leader_election.cc:302] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816ec746c35042278bbb9794c9c4f38f, dfcfb5da82334208bcc3e5a82db4ef76; no voters:
I20260927 11:21:54.113610 18386 raft_consensus.cc:2781] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:21:54.113667 18386 raft_consensus.cc:482] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:21:54.113701 18386 raft_consensus.cc:3037] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:54.114403 18386 raft_consensus.cc:504] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcfb5da82334208bcc3e5a82db4ef76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36825 } } peers { permanent_uuid: "63eff2629e9c49d09f26fff405c0b2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35221 } } peers { permanent_uuid: "816ec746c35042278bbb9794c9c4f38f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33747 } }
I20260927 11:21:54.114555 18386 leader_election.cc:288] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [CANDIDATE]: Term 1 election: Requested vote from peers dfcfb5da82334208bcc3e5a82db4ef76 (127.0.98.3:36825), 63eff2629e9c49d09f26fff405c0b2f3 (127.0.98.2:35221)
I20260927 11:21:54.114702 18319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7e7a1abe104ca2b3193b204f07e3e1" candidate_uuid: "816ec746c35042278bbb9794c9c4f38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcfb5da82334208bcc3e5a82db4ef76"
I20260927 11:21:54.114770 18188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7e7a1abe104ca2b3193b204f07e3e1" candidate_uuid: "816ec746c35042278bbb9794c9c4f38f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63eff2629e9c49d09f26fff405c0b2f3"
I20260927 11:21:54.114801 18319 raft_consensus.cc:3037] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:54.114837 18188 raft_consensus.cc:3037] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:54.115710 18188 raft_consensus.cc:2454] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816ec746c35042278bbb9794c9c4f38f in term 1.
I20260927 11:21:54.115710 18319 raft_consensus.cc:2454] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 816ec746c35042278bbb9794c9c4f38f in term 1.
I20260927 11:21:54.115912 17993 leader_election.cc:302] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 816ec746c35042278bbb9794c9c4f38f, dfcfb5da82334208bcc3e5a82db4ef76; no voters:
I20260927 11:21:54.116017 18386 raft_consensus.cc:2781] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:21:54.116163 18386 raft_consensus.cc:686] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [term 1 LEADER]: Becoming Leader. State: Replica: 816ec746c35042278bbb9794c9c4f38f, State: Running, Role: LEADER
I20260927 11:21:54.116269 18386 consensus_queue.cc:237] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcfb5da82334208bcc3e5a82db4ef76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36825 } } peers { permanent_uuid: "63eff2629e9c49d09f26fff405c0b2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35221 } } peers { permanent_uuid: "816ec746c35042278bbb9794c9c4f38f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33747 } }
W20260927 11:21:54.117166 18366 tablet.cc:2369] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:21:54.117240 17914 catalog_manager.cc:5495] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f reported cstate change: term changed from 0 to 1, leader changed from <none> to 816ec746c35042278bbb9794c9c4f38f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "816ec746c35042278bbb9794c9c4f38f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcfb5da82334208bcc3e5a82db4ef76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63eff2629e9c49d09f26fff405c0b2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "816ec746c35042278bbb9794c9c4f38f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33747 } health_report { overall_health: HEALTHY } } }
I20260927 11:21:54.123050 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:54.120666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42838 (local address 127.0.98.1:33747)
0927 11:21:54.120746 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:54.120747 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:54.120758 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:21:54.120790 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:54.120790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:54.120812 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:54.120904 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:21:54.120997 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:54.121512 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:54.122240 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:54.122464 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:54.122527 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:54.122650 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:21:54.122749 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:21:54.122809 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:54.122855 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 11:21:54.122873 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 11:21:54.125097 18319 raft_consensus.cc:1261] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76 [term 1 FOLLOWER]: Refusing update from remote peer 816ec746c35042278bbb9794c9c4f38f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:54.125142 18188 raft_consensus.cc:1261] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3 [term 1 FOLLOWER]: Refusing update from remote peer 816ec746c35042278bbb9794c9c4f38f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:54.125540 18386 consensus_queue.cc:1035] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcfb5da82334208bcc3e5a82db4ef76" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:54.125705 18389 consensus_queue.cc:1035] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 816ec746c35042278bbb9794c9c4f38f [LEADER]: Connected to new peer: Peer: permanent_uuid: "63eff2629e9c49d09f26fff405c0b2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:54.128087 18392 mvcc.cc:205] Tried to move back new op lower bound from 7333921235452940288 to 7333921235421102080. Current Snapshot: MvccSnapshot[applied={T|T < 7333921235452940288}]
I20260927 11:21:54.128376 18394 mvcc.cc:205] Tried to move back new op lower bound from 7333921235452940288 to 7333921235421102080. Current Snapshot: MvccSnapshot[applied={T|T < 7333921235452940288}]
I20260927 11:21:54.128482 18393 mvcc.cc:205] Tried to move back new op lower bound from 7333921235452940288 to 7333921235421102080. Current Snapshot: MvccSnapshot[applied={T|T < 7333921235452940288}]
I20260927 11:21:54.130427 17915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:54.130529 17915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:54.131265 17915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fc3405fea9343c99d794d922d1a18ca: Sending DeleteTablet for 3 replicas of tablet 6c7e7a1abe104ca2b3193b204f07e3e1
I20260927 11:21:54.131608 18168 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7e7a1abe104ca2b3193b204f07e3e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:54 UTC) from {username='slave'} at 127.0.0.1:56758
I20260927 11:21:54.131678 18299 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7e7a1abe104ca2b3193b204f07e3e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:54 UTC) from {username='slave'} at 127.0.0.1:51682
I20260927 11:21:54.131845 18402 tablet_replica.cc:330] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3: stopping tablet replica
I20260927 11:21:54.131888 18037 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7e7a1abe104ca2b3193b204f07e3e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:54 UTC) from {username='slave'} at 127.0.0.1:42836
I20260927 11:21:54.131940 18402 raft_consensus.cc:2229] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:21:54.132038 18402 raft_consensus.cc:2258] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:54.132288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 17975
I20260927 11:21:54.132500 18402 ts_tablet_manager.cc:1905] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:54.133992 18403 tablet_replica.cc:330] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76: stopping tablet replica
I20260927 11:21:54.134102 18403 raft_consensus.cc:2229] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:21:54.134214 18403 raft_consensus.cc:2258] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:54.134357 18402 ts_tablet_manager.cc:1918] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:54.134433 18402 log.cc:1196] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/6c7e7a1abe104ca2b3193b204f07e3e1
I20260927 11:21:54.134639 18403 ts_tablet_manager.cc:1905] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:54.134747 18402 ts_tablet_manager.cc:1939] T 6c7e7a1abe104ca2b3193b204f07e3e1 P 63eff2629e9c49d09f26fff405c0b2f3: Deleting consensus metadata
I20260927 11:21:54.135212 17901 catalog_manager.cc:4841] TS 63eff2629e9c49d09f26fff405c0b2f3 (127.0.98.2:35221): tablet 6c7e7a1abe104ca2b3193b204f07e3e1 (table test-table [id=96fef1b4158a45ec8fa8e5da59ae7886]) successfully deleted
I20260927 11:21:54.136508 18403 ts_tablet_manager.cc:1918] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:54.136584 18403 log.cc:1196] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/6c7e7a1abe104ca2b3193b204f07e3e1
I20260927 11:21:54.136816 18403 ts_tablet_manager.cc:1939] T 6c7e7a1abe104ca2b3193b204f07e3e1 P dfcfb5da82334208bcc3e5a82db4ef76: Deleting consensus metadata
I20260927 11:21:54.137311 17902 catalog_manager.cc:4841] TS dfcfb5da82334208bcc3e5a82db4ef76 (127.0.98.3:36825): tablet 6c7e7a1abe104ca2b3193b204f07e3e1 (table test-table [id=96fef1b4158a45ec8fa8e5da59ae7886]) successfully deleted
W20260927 11:21:54.138964 17902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33747 (error 108)
I20260927 11:21:54.139159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 18106
I20260927 11:21:54.139448 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:54.139332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33747 (local address 127.0.0.1:42850)
0927 11:21:54.139378 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:54.139410 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 11:21:54.139557 17902 catalog_manager.cc:4584] TS 816ec746c35042278bbb9794c9c4f38f (127.0.98.1:33747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c7e7a1abe104ca2b3193b204f07e3e1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33747: connect: Connection refused (error 111)
I20260927 11:21:54.144709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 18237
I20260927 11:21:54.149185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 17884
2026-09-27T11:21:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (861 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:21:54 dist-test-slave-lsvg krb5kdc[18408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:21:54 dist-test-slave-lsvg krb5kdc[18408](info): set up 2 sockets
Sep 27 11:21:54 dist-test-slave-lsvg krb5kdc[18408](info): commencing operation
krb5kdc: starting...
W20260927 11:21:55.200883 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:21:55 dist-test-slave-lsvg krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:21:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:21:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:21:55.320356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41001
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:55.402081 18421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:55.402297 18421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:55.402320 18421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:55.403416 18421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:21:55.403451 18421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:55.403466 18421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:55.403478 18421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:21:55.403491 18421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:21:55.404872 18421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:55.405136 18421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:55.405324 18421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:21:55.410331 18421 server_base.cc:1034] running on GCE node
W20260927 11:21:55.410311 18427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:55.410315 18426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:55.410293 18429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:55.410843 18421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:55.411166 18421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:55.412338 18421 hybrid_clock.cc:648] HybridClock initialized: now 1790508115412332 us; error 40 us; skew 500 ppm
Sep 27 11:21:55 dist-test-slave-lsvg krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:55.414924 18421 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:21:55.415457 18421 webserver.cc:460] Webserver started at http://127.0.98.62:41437/ using document root <none> and password file <none>
I20260927 11:21:55.415697 18421 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:55.415766 18421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:55.415910 18421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:55.416940 18421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "e9bd881691814acdbe165fbec476fbbb"
format_stamp: "Formatted at 2026-09-27 11:21:55 on dist-test-slave-lsvg"
server_key: "47ceedf91c01e647a2b94fb35a535a9d"
server_key_iv: "4bdbf8099489361c799e44d26198a796"
server_key_version: "encryptionkey@0"
I20260927 11:21:55.417371 18421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9bd881691814acdbe165fbec476fbbb"
format_stamp: "Formatted at 2026-09-27 11:21:55 on dist-test-slave-lsvg"
server_key: "47ceedf91c01e647a2b94fb35a535a9d"
server_key_iv: "4bdbf8099489361c799e44d26198a796"
server_key_version: "encryptionkey@0"
I20260927 11:21:55.418684 18421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:55.419391 18436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:55.419612 18421 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:55.419718 18421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "e9bd881691814acdbe165fbec476fbbb"
format_stamp: "Formatted at 2026-09-27 11:21:55 on dist-test-slave-lsvg"
server_key: "47ceedf91c01e647a2b94fb35a535a9d"
server_key_iv: "4bdbf8099489361c799e44d26198a796"
server_key_version: "encryptionkey@0"
I20260927 11:21:55.419791 18421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:55.436640 18421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:55.439141 18421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:55.439311 18421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:55.443336 18421 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41001
I20260927 11:21:55.443368 18488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41001 every 8 connection(s)
I20260927 11:21:55.443723 18421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:21:55.444559 18489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:55.445330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 18421
I20260927 11:21:55.445449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:21:55.445657 392 external_mini_cluster.cc:1428] Setting key 6de4c7d3362bcc6d88936599707970b7
I20260927 11:21:55.447777 18489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb: Bootstrap starting.
I20260927 11:21:55.448367 18489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:55.448668 18489 log.cc:824] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:55.449404 18489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb: No bootstrap required, opened a new log
I20260927 11:21:55.451478 18489 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bd881691814acdbe165fbec476fbbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41001 } }
I20260927 11:21:55.451649 18489 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:55.451684 18489 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9bd881691814acdbe165fbec476fbbb, State: Initialized, Role: FOLLOWER
I20260927 11:21:55.451817 18489 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bd881691814acdbe165fbec476fbbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41001 } }
I20260927 11:21:55.451891 18489 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:21:55.451936 18489 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:21:55.451987 18489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [term 0 FOLLOWER]: Advancing to term 1
Sep 27 11:21:55 dist-test-slave-lsvg krb5kdc[18408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:21:55.452708 18489 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bd881691814acdbe165fbec476fbbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41001 } }
I20260927 11:21:55.452821 18489 leader_election.cc:302] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9bd881691814acdbe165fbec476fbbb; no voters:
I20260927 11:21:55.453035 18489 leader_election.cc:288] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:21:55.453082 18494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:21:55.453194 18494 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [term 1 LEADER]: Becoming Leader. State: Replica: e9bd881691814acdbe165fbec476fbbb, State: Running, Role: LEADER
I20260927 11:21:55.453322 18489 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:21:55.453338 18494 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bd881691814acdbe165fbec476fbbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41001 } }
I20260927 11:21:55.453696 18494 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9bd881691814acdbe165fbec476fbbb. Latest consensus state: current_term: 1 leader_uuid: "e9bd881691814acdbe165fbec476fbbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bd881691814acdbe165fbec476fbbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41001 } } }
I20260927 11:21:55.453711 18495 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9bd881691814acdbe165fbec476fbbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bd881691814acdbe165fbec476fbbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41001 } } }
I20260927 11:21:55.453889 18494 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [sys.catalog]: This master's current role is: LEADER
I20260927 11:21:55.453903 18495 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb [sys.catalog]: This master's current role is: LEADER
W20260927 11:21:55.455188 18509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:21:55.455251 18509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:21:55.455309 18503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:21:55.455513 18503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:21:55.457134 18503 catalog_manager.cc:1347] Generated new cluster ID: 69b249271cd24028affb3467fded0a8d
I20260927 11:21:55.457192 18503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:21:55.457463 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.446182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.0.98.62:41001)
0927 11:21:55.446356 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:55.446364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:55.446389 (+ 25us) server_negotiation.cc:407] Connection header received
0927 11:21:55.446774 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:55.446781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:55.446823 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:55.446939 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:55.447764 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.448753 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.449601 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.450201 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.450321 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.452877 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:55.452882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:55.452882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:55.452902 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:55.455539 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:55.455994 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:55.455997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:55.456002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:55.456120 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:55.456359 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:55.456360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:55.456361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:55.456670 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:55.456844 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:55.456979 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 11:21:55.457269 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260927 11:21:55.465916 18503 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:21:55.466435 18503 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:21:55.478998 18503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb: Generated new TSK 0
I20260927 11:21:55.479223 18503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:21:55.512256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41001
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:55.594177 18515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:55.594378 18515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:55.594400 18515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:55.595481 18515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:55.595520 18515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:55.595546 18515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:21:55.597177 18515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:55.597363 18515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:55.597546 18515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:55.602427 18520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:55.602439 18521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:55.602445 18523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:55.602532 18515 server_base.cc:1034] running on GCE node
I20260927 11:21:55.602847 18515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:55.603044 18515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:55.604180 18515 hybrid_clock.cc:648] HybridClock initialized: now 1790508115604168 us; error 29 us; skew 500 ppm
Sep 27 11:21:55 dist-test-slave-lsvg krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:55.606745 18515 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:21:55.607205 18515 webserver.cc:460] Webserver started at http://127.0.98.1:45113/ using document root <none> and password file <none>
I20260927 11:21:55.607446 18515 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:55.607491 18515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:55.607615 18515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:55.608651 18515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "da2dcebc900e469fb7d8337a8ab97251"
format_stamp: "Formatted at 2026-09-27 11:21:55 on dist-test-slave-lsvg"
server_key: "b9c3e69fe5957538e329eaa87d343ff5"
server_key_iv: "16b890651e6d53b7fb409ef205bf6b61"
server_key_version: "encryptionkey@0"
I20260927 11:21:55.609004 18515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "da2dcebc900e469fb7d8337a8ab97251"
format_stamp: "Formatted at 2026-09-27 11:21:55 on dist-test-slave-lsvg"
server_key: "b9c3e69fe5957538e329eaa87d343ff5"
server_key_iv: "16b890651e6d53b7fb409ef205bf6b61"
server_key_version: "encryptionkey@0"
I20260927 11:21:55.610376 18515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:55.611205 18530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:55.611407 18515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:55.611500 18515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "da2dcebc900e469fb7d8337a8ab97251"
format_stamp: "Formatted at 2026-09-27 11:21:55 on dist-test-slave-lsvg"
server_key: "b9c3e69fe5957538e329eaa87d343ff5"
server_key_iv: "16b890651e6d53b7fb409ef205bf6b61"
server_key_version: "encryptionkey@0"
I20260927 11:21:55.611577 18515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:55.651723 18515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:55.654083 18515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:55.654307 18515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:55.654572 18515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:55.654913 18515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:55.654966 18515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:55.655007 18515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:55.655036 18515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:55.661252 18515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41819
I20260927 11:21:55.661295 18643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41819 every 8 connection(s)
I20260927 11:21:55.661638 18515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:21:55.667467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 18515
I20260927 11:21:55.667577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:21:55.667784 392 external_mini_cluster.cc:1428] Setting key 93e9ccb5cfbf5f12c903c082571e15df
Sep 27 11:21:55 dist-test-slave-lsvg krb5kdc[18408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:21:55.671532 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.662375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45651 (local address 127.0.98.62:41001)
0927 11:21:55.662437 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:55.662439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:55.662673 (+ 234us) server_negotiation.cc:407] Connection header received
0927 11:21:55.663660 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:55.663661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:55.663672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:55.663734 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:55.664339 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.664842 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.665940 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.666151 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.666207 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.668571 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:55.668573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:55.668573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:55.668581 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:55.670127 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:55.670532 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:55.670533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:55.670534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:55.670602 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:55.670868 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:55.670869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:55.670870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:55.670985 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:55.671060 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:55.671353 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 11:21:55.671441 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 11:21:55.671742 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.662369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41001 (local address 127.0.98.1:45651)
0927 11:21:55.662537 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:55.662582 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:55.663554 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:55.663789 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:55.663792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:55.664065 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:55.664267 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:55.664271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.664952 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.664953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:55.665838 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:55.665841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.666232 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.666232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:55.666336 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.666839 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:55.666848 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:55.668481 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:55.670213 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:55.670221 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:55.670223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:55.670451 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:55.670679 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:55.670680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:55.670680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:55.670793 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:55.671053 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:55.671060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:55.671215 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 11:21:55.671323 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 11:21:55.671485 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260927 11:21:55.672259 18644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41001
I20260927 11:21:55.672372 18644 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:55.672583 18644 heartbeater.cc:511] Master 127.0.98.62:41001 requested a full tablet report, sending...
I20260927 11:21:55.673216 18453 ts_manager.cc:194] Registered new tserver with Master: da2dcebc900e469fb7d8337a8ab97251 (127.0.98.1:41819)
I20260927 11:21:55.674189 18453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45651
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:21:55.694008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41001
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:55.774914 18649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:55.775110 18649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:55.775131 18649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:55.776302 18649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:55.776343 18649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:55.776369 18649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:21:55.777899 18649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:55.778151 18649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:55.778426 18649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:55.783488 18654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:55.783565 18649 server_base.cc:1034] running on GCE node
W20260927 11:21:55.783519 18657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:55.783457 18655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:55.783859 18649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:55.784070 18649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:55.785231 18649 hybrid_clock.cc:648] HybridClock initialized: now 1790508115785213 us; error 33 us; skew 500 ppm
Sep 27 11:21:55 dist-test-slave-lsvg krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:55.787683 18649 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:21:55.788179 18649 webserver.cc:460] Webserver started at http://127.0.98.2:46343/ using document root <none> and password file <none>
I20260927 11:21:55.788401 18649 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:55.788486 18649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:55.788633 18649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:55.789745 18649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "0a53aba59dda4fe78d9f09e4b5c7babc"
format_stamp: "Formatted at 2026-09-27 11:21:55 on dist-test-slave-lsvg"
server_key: "f27c669c8b377f9996b3c2f429ca1fe0"
server_key_iv: "b319cb60d4e51ca796adcdf01fe7649b"
server_key_version: "encryptionkey@0"
I20260927 11:21:55.790109 18649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0a53aba59dda4fe78d9f09e4b5c7babc"
format_stamp: "Formatted at 2026-09-27 11:21:55 on dist-test-slave-lsvg"
server_key: "f27c669c8b377f9996b3c2f429ca1fe0"
server_key_iv: "b319cb60d4e51ca796adcdf01fe7649b"
server_key_version: "encryptionkey@0"
I20260927 11:21:55.791384 18649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:21:55.792090 18664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:55.792264 18649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:21:55.792366 18649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "0a53aba59dda4fe78d9f09e4b5c7babc"
format_stamp: "Formatted at 2026-09-27 11:21:55 on dist-test-slave-lsvg"
server_key: "f27c669c8b377f9996b3c2f429ca1fe0"
server_key_iv: "b319cb60d4e51ca796adcdf01fe7649b"
server_key_version: "encryptionkey@0"
I20260927 11:21:55.792438 18649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:55.801561 18649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:55.803879 18649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:55.804049 18649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:55.804287 18649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:55.804603 18649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:55.804656 18649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:55.804689 18649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:55.804716 18649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:55.810963 18649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45365
I20260927 11:21:55.811013 18777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45365 every 8 connection(s)
I20260927 11:21:55.811342 18649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:21:55 dist-test-slave-lsvg krb5kdc[18408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:21:55.819000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 18649
I20260927 11:21:55.819133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:21:55.819396 392 external_mini_cluster.cc:1428] Setting key d8564cb6a11d55b3bc99e8de03e035ca
I20260927 11:21:55.820760 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.812086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52037 (local address 127.0.98.62:41001)
0927 11:21:55.812152 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:55.812153 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:55.812335 (+ 182us) server_negotiation.cc:407] Connection header received
0927 11:21:55.813267 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:55.813268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:55.813277 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:55.813337 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:55.813911 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.814456 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.815444 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.815652 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.815706 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.817930 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:55.817932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:55.817932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:55.817938 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:55.819190 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:55.819713 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:55.819715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:55.819715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:55.819780 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:55.820055 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:55.820056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:55.820057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:55.820209 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:55.820288 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:55.820583 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 11:21:55.820670 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 11:21:55.820981 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.812056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41001 (local address 127.0.98.2:52037)
0927 11:21:55.812206 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:55.812247 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:55.813178 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:55.813393 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:55.813396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:55.813638 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:55.813832 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:55.813836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.814538 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.814538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:55.815355 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:55.815357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.815726 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.815727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:55.815825 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.816412 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:55.816420 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:55.817839 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:55.819291 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:55.819298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:55.819300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:55.819619 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:55.819842 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:55.819843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:55.819843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:55.819951 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:55.820279 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:55.820285 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:55.820429 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 11:21:55.820562 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 11:21:55.820722 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260927 11:21:55.821516 18778 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41001
I20260927 11:21:55.821632 18778 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:55.821831 18778 heartbeater.cc:511] Master 127.0.98.62:41001 requested a full tablet report, sending...
I20260927 11:21:55.822252 18453 ts_manager.cc:194] Registered new tserver with Master: 0a53aba59dda4fe78d9f09e4b5c7babc (127.0.98.2:45365)
I20260927 11:21:55.822777 18453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52037
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:21:55.843802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41001
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:55.924917 18783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:55.925225 18783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:55.925271 18783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:55.926465 18783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:55.926535 18783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:55.926579 18783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:21:55.928061 18783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:55.928335 18783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:55.928578 18783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:55.933413 18791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:55.933413 18788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:55.933461 18789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:55.933941 18783 server_base.cc:1034] running on GCE node
I20260927 11:21:55.934172 18783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:55.934435 18783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:55.935602 18783 hybrid_clock.cc:648] HybridClock initialized: now 1790508115935591 us; error 26 us; skew 500 ppm
Sep 27 11:21:55 dist-test-slave-lsvg krb5kdc[18408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:55.938164 18783 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:21:55.938652 18783 webserver.cc:460] Webserver started at http://127.0.98.3:46651/ using document root <none> and password file <none>
I20260927 11:21:55.938879 18783 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:55.938944 18783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:55.939116 18783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:55.940150 18783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "40e79c73db8f471e9fc1575ff5fd4dad"
format_stamp: "Formatted at 2026-09-27 11:21:55 on dist-test-slave-lsvg"
server_key: "429851f894bdcb62a94028279c461464"
server_key_iv: "6a48519ee51917abffebf81996d52d87"
server_key_version: "encryptionkey@0"
I20260927 11:21:55.940512 18783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "40e79c73db8f471e9fc1575ff5fd4dad"
format_stamp: "Formatted at 2026-09-27 11:21:55 on dist-test-slave-lsvg"
server_key: "429851f894bdcb62a94028279c461464"
server_key_iv: "6a48519ee51917abffebf81996d52d87"
server_key_version: "encryptionkey@0"
I20260927 11:21:55.941892 18783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:21:55.942584 18798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:55.942762 18783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:21:55.942862 18783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "40e79c73db8f471e9fc1575ff5fd4dad"
format_stamp: "Formatted at 2026-09-27 11:21:55 on dist-test-slave-lsvg"
server_key: "429851f894bdcb62a94028279c461464"
server_key_iv: "6a48519ee51917abffebf81996d52d87"
server_key_version: "encryptionkey@0"
I20260927 11:21:55.942945 18783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:55.955382 18783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:55.957268 18783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:55.957422 18783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:55.957640 18783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:55.957937 18783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:55.957991 18783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:55.958033 18783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:55.958060 18783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:55.963810 18783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34275
I20260927 11:21:55.963850 18911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34275 every 8 connection(s)
I20260927 11:21:55.964203 18783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:21:55.968521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 18783
I20260927 11:21:55.968626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:21:55.968791 392 external_mini_cluster.cc:1428] Setting key 68b27bd2be97e148836a020db66c3e4e
Sep 27 11:21:55 dist-test-slave-lsvg krb5kdc[18408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:21:55.973655 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.964946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37139 (local address 127.0.98.62:41001)
0927 11:21:55.965032 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:55.965034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:55.965211 (+ 177us) server_negotiation.cc:407] Connection header received
0927 11:21:55.966182 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:55.966183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:55.966197 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:55.966249 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:55.966884 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.967387 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.968436 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.968734 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.968822 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.971088 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:55.971090 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:55.971090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:55.971098 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:55.972253 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:55.972658 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:55.972659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:55.972659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:55.972719 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:55.972959 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:55.972960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:55.972961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:55.973131 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:55.973212 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:55.973482 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 11:21:55.973562 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 11:21:55.973836 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.964856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41001 (local address 127.0.98.3:37139)
0927 11:21:55.965083 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:55.965130 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:55.966059 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:55.966315 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:55.966319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:55.966602 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:55.966786 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:55.966790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.967486 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.967487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:55.968289 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:55.968291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.968831 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.968832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:55.968993 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.969569 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:55.969579 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:55.970994 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:55.972345 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:55.972351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:55.972353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:55.972574 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:55.972788 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:55.972789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:55.972789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:55.972893 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:55.973202 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:55.973208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:55.973348 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 11:21:55.973449 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 11:21:55.973608 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
I20260927 11:21:55.974273 18912 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41001
I20260927 11:21:55.974365 18912 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:55.974540 18912 heartbeater.cc:511] Master 127.0.98.62:41001 requested a full tablet report, sending...
I20260927 11:21:55.974900 18453 ts_manager.cc:194] Registered new tserver with Master: 40e79c73db8f471e9fc1575ff5fd4dad (127.0.98.3:34275)
I20260927 11:21:55.975221 18453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37139
I20260927 11:21:55.980513 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:21:55.987823 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.981807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.0.98.62:41001)
0927 11:21:55.981890 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:55.981892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:55.981944 (+ 52us) server_negotiation.cc:407] Connection header received
0927 11:21:55.981996 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:55.981997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:55.982006 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:55.982053 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:55.982564 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.983026 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.983703 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.983878 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.983928 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.985380 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:55.985381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:55.985382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:55.985388 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:55.986739 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:55.987122 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:55.987123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:55.987123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:55.987173 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:55.987353 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:55.987354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:55.987354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:55.987473 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:55.987549 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:55.987651 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 11:21:55.987700 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 11:21:55.989357 18453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:21:55.990840 18453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:21:55.997642 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.993176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45365 (local address 127.0.0.1:46672)
0927 11:21:55.993465 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:55.993474 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:55.993522 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:55.993836 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:55.993839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:55.993841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:55.993920 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:55.993922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.995077 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.995078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:55.996312 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:55.996314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.997245 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.997246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:55.997330 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.997348 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 11:21:55.997415 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 11:21:55.997495 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260927 11:21:55.997727 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.993601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.0.98.3:34275)
0927 11:21:55.993780 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:55.993784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:55.993794 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:21:55.993847 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:55.993849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:55.993864 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:55.993945 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:55.994171 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.995352 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.996648 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.997289 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.997355 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.997414 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:55.997589 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 11:21:55.997636 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260927 11:21:55.997756 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.993347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.0.98.2:45365)
0927 11:21:55.993656 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:55.993661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:55.993673 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:21:55.993715 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:55.993718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:55.993734 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:55.993811 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:55.994007 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.994928 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.996395 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.997161 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.997235 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.997294 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:55.997626 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 11:21:55.997676 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":89,"threads_started":1}
I20260927 11:21:55.998034 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.993129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41819 (local address 127.0.0.1:60636)
0927 11:21:55.993311 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:55.993355 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:55.993426 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:55.994212 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:55.994213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:55.994215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:55.994260 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:55.994261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.995505 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.995506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:55.996590 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:55.996592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.997784 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.997785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:55.997849 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.997852 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:21:55.997930 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 11:21:55.997968 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":61,"threads_started":1}
I20260927 11:21:55.998066 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.993229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.0.98.1:41819)
0927 11:21:55.993982 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:55.993987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:55.993997 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:21:55.994035 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:55.994038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:55.994053 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:55.994125 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:55.994347 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.995422 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.996692 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.997632 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.997692 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.997749 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:55.997949 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 11:21:55.997992 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":72,"threads_started":1}
I20260927 11:21:55.998317 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:55.993339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34275 (local address 127.0.0.1:38640)
0927 11:21:55.993539 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:55.993548 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:55.993588 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:55.994033 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:55.994034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:55.994036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:55.994089 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:55.994091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.995457 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.995457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:55.996549 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:55.996551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:55.997370 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:55.997371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:55.997434 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:55.997438 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:21:55.997489 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 11:21:55.997538 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260927 11:21:55.998888 18712 tablet_service.cc:1467] Processing CreateTablet for tablet 779d6fdff4f4428f825b05c84c9046e5 (DEFAULT_TABLE table=test-table [id=c124dbec80df4095b75a8440767e367d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:55.998912 18578 tablet_service.cc:1467] Processing CreateTablet for tablet 779d6fdff4f4428f825b05c84c9046e5 (DEFAULT_TABLE table=test-table [id=c124dbec80df4095b75a8440767e367d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:55.999267 18578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779d6fdff4f4428f825b05c84c9046e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:55.999267 18712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779d6fdff4f4428f825b05c84c9046e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:55.999194 18846 tablet_service.cc:1467] Processing CreateTablet for tablet 779d6fdff4f4428f825b05c84c9046e5 (DEFAULT_TABLE table=test-table [id=c124dbec80df4095b75a8440767e367d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:55.999449 18846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 779d6fdff4f4428f825b05c84c9046e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:56.001973 18929 tablet_bootstrap.cc:492] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad: Bootstrap starting.
I20260927 11:21:56.002549 18931 tablet_bootstrap.cc:492] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc: Bootstrap starting.
I20260927 11:21:56.002671 18929 tablet_bootstrap.cc:654] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:56.003243 18931 tablet_bootstrap.cc:654] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:56.003253 18929 log.cc:824] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:56.003803 18931 log.cc:824] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:56.004065 18929 tablet_bootstrap.cc:492] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad: No bootstrap required, opened a new log
I20260927 11:21:56.004143 18929 ts_tablet_manager.cc:1397] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:21:56.004166 18930 tablet_bootstrap.cc:492] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251: Bootstrap starting.
I20260927 11:21:56.004557 18931 tablet_bootstrap.cc:492] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc: No bootstrap required, opened a new log
I20260927 11:21:56.004633 18931 ts_tablet_manager.cc:1397] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:21:56.005584 18930 tablet_bootstrap.cc:654] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:56.006273 18930 log.cc:824] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:56.006217 18929 raft_consensus.cc:348] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da2dcebc900e469fb7d8337a8ab97251" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41819 } } peers { permanent_uuid: "0a53aba59dda4fe78d9f09e4b5c7babc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45365 } } peers { permanent_uuid: "40e79c73db8f471e9fc1575ff5fd4dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34275 } }
I20260927 11:21:56.006410 18929 raft_consensus.cc:374] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:56.006488 18929 raft_consensus.cc:729] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e79c73db8f471e9fc1575ff5fd4dad, State: Initialized, Role: FOLLOWER
I20260927 11:21:56.006615 18931 raft_consensus.cc:348] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da2dcebc900e469fb7d8337a8ab97251" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41819 } } peers { permanent_uuid: "0a53aba59dda4fe78d9f09e4b5c7babc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45365 } } peers { permanent_uuid: "40e79c73db8f471e9fc1575ff5fd4dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34275 } }
I20260927 11:21:56.006703 18929 consensus_queue.cc:260] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da2dcebc900e469fb7d8337a8ab97251" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41819 } } peers { permanent_uuid: "0a53aba59dda4fe78d9f09e4b5c7babc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45365 } } peers { permanent_uuid: "40e79c73db8f471e9fc1575ff5fd4dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34275 } }
I20260927 11:21:56.006817 18931 raft_consensus.cc:374] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:56.006888 18931 raft_consensus.cc:729] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a53aba59dda4fe78d9f09e4b5c7babc, State: Initialized, Role: FOLLOWER
I20260927 11:21:56.006981 18929 ts_tablet_manager.cc:1428] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:21:56.007053 18930 tablet_bootstrap.cc:492] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251: No bootstrap required, opened a new log
I20260927 11:21:56.007124 18912 heartbeater.cc:503] Master 127.0.98.62:41001 was elected leader, sending a full tablet report...
I20260927 11:21:56.007092 18931 consensus_queue.cc:260] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da2dcebc900e469fb7d8337a8ab97251" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41819 } } peers { permanent_uuid: "0a53aba59dda4fe78d9f09e4b5c7babc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45365 } } peers { permanent_uuid: "40e79c73db8f471e9fc1575ff5fd4dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34275 } }
I20260927 11:21:56.007164 18930 ts_tablet_manager.cc:1397] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 11:21:56.007370 18778 heartbeater.cc:503] Master 127.0.98.62:41001 was elected leader, sending a full tablet report...
I20260927 11:21:56.007380 18931 ts_tablet_manager.cc:1428] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:21:56.008986 18930 raft_consensus.cc:348] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da2dcebc900e469fb7d8337a8ab97251" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41819 } } peers { permanent_uuid: "0a53aba59dda4fe78d9f09e4b5c7babc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45365 } } peers { permanent_uuid: "40e79c73db8f471e9fc1575ff5fd4dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34275 } }
I20260927 11:21:56.009209 18930 raft_consensus.cc:374] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:56.009238 18930 raft_consensus.cc:729] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da2dcebc900e469fb7d8337a8ab97251, State: Initialized, Role: FOLLOWER
I20260927 11:21:56.009346 18930 consensus_queue.cc:260] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da2dcebc900e469fb7d8337a8ab97251" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41819 } } peers { permanent_uuid: "0a53aba59dda4fe78d9f09e4b5c7babc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45365 } } peers { permanent_uuid: "40e79c73db8f471e9fc1575ff5fd4dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34275 } }
I20260927 11:21:56.009618 18930 ts_tablet_manager.cc:1428] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:21:56.009648 18644 heartbeater.cc:503] Master 127.0.98.62:41001 was elected leader, sending a full tablet report...
W20260927 11:21:56.061887 18779 tablet.cc:2369] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:21:56.089970 18937 raft_consensus.cc:482] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:21:56.090147 18937 raft_consensus.cc:504] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da2dcebc900e469fb7d8337a8ab97251" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41819 } } peers { permanent_uuid: "0a53aba59dda4fe78d9f09e4b5c7babc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45365 } } peers { permanent_uuid: "40e79c73db8f471e9fc1575ff5fd4dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34275 } }
I20260927 11:21:56.090624 18937 leader_election.cc:288] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a53aba59dda4fe78d9f09e4b5c7babc (127.0.98.2:45365), 40e79c73db8f471e9fc1575ff5fd4dad (127.0.98.3:34275)
I20260927 11:21:56.094758 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.090898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55115 (local address 127.0.98.2:45365)
0927 11:21:56.090949 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.090951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.090964 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:21:56.091042 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.091043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.091058 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.091138 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:56.091290 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.092060 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.093770 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.094346 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.094431 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.094443 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.094642 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.094687 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 11:21:56.094758 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.090936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47537 (local address 127.0.98.3:34275)
0927 11:21:56.090995 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.090997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.091200 (+ 203us) server_negotiation.cc:407] Connection header received
0927 11:21:56.091229 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.091230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.091242 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.091305 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:56.091485 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.092169 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.093769 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.094359 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.094431 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.094443 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.094641 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.094687 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 11:21:56.094761 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.090806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45365 (local address 127.0.98.1:55115)
0927 11:21:56.090872 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:56.090885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:56.090954 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:56.091148 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:56.091148 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:56.091150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:56.091212 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.091214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.092157 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.092158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.093657 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.093658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.094445 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.094446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.094535 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.094540 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:21:56.094608 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 11:21:56.094659 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":20}
I20260927 11:21:56.094770 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.090850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34275 (local address 127.0.98.1:47537)
0927 11:21:56.091054 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:56.091066 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:56.091130 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:56.091334 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:56.091335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:56.091336 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:56.091389 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.091391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.092256 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.092256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.093650 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.093654 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.094477 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.094478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.094559 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.094563 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:21:56.094622 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 11:21:56.094659 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":37,"thread_start_us":137,"threads_started":1}
I20260927 11:21:56.095203 18732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779d6fdff4f4428f825b05c84c9046e5" candidate_uuid: "da2dcebc900e469fb7d8337a8ab97251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a53aba59dda4fe78d9f09e4b5c7babc" is_pre_election: true
I20260927 11:21:56.095206 18866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779d6fdff4f4428f825b05c84c9046e5" candidate_uuid: "da2dcebc900e469fb7d8337a8ab97251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e79c73db8f471e9fc1575ff5fd4dad" is_pre_election: true
I20260927 11:21:56.095376 18732 raft_consensus.cc:2454] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da2dcebc900e469fb7d8337a8ab97251 in term 0.
I20260927 11:21:56.095376 18866 raft_consensus.cc:2454] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da2dcebc900e469fb7d8337a8ab97251 in term 0.
I20260927 11:21:56.095646 18533 leader_election.cc:302] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40e79c73db8f471e9fc1575ff5fd4dad, da2dcebc900e469fb7d8337a8ab97251; no voters:
I20260927 11:21:56.095814 18937 raft_consensus.cc:2781] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:21:56.095891 18937 raft_consensus.cc:482] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:21:56.095909 18937 raft_consensus.cc:3037] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:56.096653 18937 raft_consensus.cc:504] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da2dcebc900e469fb7d8337a8ab97251" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41819 } } peers { permanent_uuid: "0a53aba59dda4fe78d9f09e4b5c7babc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45365 } } peers { permanent_uuid: "40e79c73db8f471e9fc1575ff5fd4dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34275 } }
I20260927 11:21:56.096796 18937 leader_election.cc:288] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [CANDIDATE]: Term 1 election: Requested vote from peers 0a53aba59dda4fe78d9f09e4b5c7babc (127.0.98.2:45365), 40e79c73db8f471e9fc1575ff5fd4dad (127.0.98.3:34275)
I20260927 11:21:56.096985 18866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779d6fdff4f4428f825b05c84c9046e5" candidate_uuid: "da2dcebc900e469fb7d8337a8ab97251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e79c73db8f471e9fc1575ff5fd4dad"
I20260927 11:21:56.097002 18732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779d6fdff4f4428f825b05c84c9046e5" candidate_uuid: "da2dcebc900e469fb7d8337a8ab97251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a53aba59dda4fe78d9f09e4b5c7babc"
I20260927 11:21:56.097069 18866 raft_consensus.cc:3037] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:56.097142 18732 raft_consensus.cc:3037] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:56.097936 18866 raft_consensus.cc:2454] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da2dcebc900e469fb7d8337a8ab97251 in term 1.
I20260927 11:21:56.098089 18533 leader_election.cc:302] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40e79c73db8f471e9fc1575ff5fd4dad, da2dcebc900e469fb7d8337a8ab97251; no voters:
I20260927 11:21:56.098140 18732 raft_consensus.cc:2454] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da2dcebc900e469fb7d8337a8ab97251 in term 1.
I20260927 11:21:56.098197 18937 raft_consensus.cc:2781] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:21:56.098390 18937 raft_consensus.cc:686] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [term 1 LEADER]: Becoming Leader. State: Replica: da2dcebc900e469fb7d8337a8ab97251, State: Running, Role: LEADER
I20260927 11:21:56.098505 18937 consensus_queue.cc:237] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da2dcebc900e469fb7d8337a8ab97251" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41819 } } peers { permanent_uuid: "0a53aba59dda4fe78d9f09e4b5c7babc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45365 } } peers { permanent_uuid: "40e79c73db8f471e9fc1575ff5fd4dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34275 } }
I20260927 11:21:56.099419 18453 catalog_manager.cc:5495] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 reported cstate change: term changed from 0 to 1, leader changed from <none> to da2dcebc900e469fb7d8337a8ab97251 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "da2dcebc900e469fb7d8337a8ab97251" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da2dcebc900e469fb7d8337a8ab97251" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a53aba59dda4fe78d9f09e4b5c7babc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40e79c73db8f471e9fc1575ff5fd4dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34275 } health_report { overall_health: UNKNOWN } } }
I20260927 11:21:56.116989 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.114416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.0.98.1:41819)
0927 11:21:56.114496 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.114498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.114511 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:21:56.114606 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.114607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.114625 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.114688 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:21:56.114842 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.115354 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.116191 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.116376 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.116468 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.116629 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:21:56.116715 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:21:56.116789 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.116900 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.116910 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 11:21:56.118916 18732 raft_consensus.cc:1261] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc [term 1 FOLLOWER]: Refusing update from remote peer da2dcebc900e469fb7d8337a8ab97251: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:56.119287 18866 raft_consensus.cc:1261] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad [term 1 FOLLOWER]: Refusing update from remote peer da2dcebc900e469fb7d8337a8ab97251: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:56.119338 18939 consensus_queue.cc:1035] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a53aba59dda4fe78d9f09e4b5c7babc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:56.119567 18937 consensus_queue.cc:1035] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e79c73db8f471e9fc1575ff5fd4dad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:56.122247 18943 mvcc.cc:205] Tried to move back new op lower bound from 7333921243619962880 to 7333921243540160512. Current Snapshot: MvccSnapshot[applied={T|T < 7333921243619962880}]
I20260927 11:21:56.123996 18944 mvcc.cc:205] Tried to move back new op lower bound from 7333921243619962880 to 7333921243540160512. Current Snapshot: MvccSnapshot[applied={T|T < 7333921243619962880 or (T in {7333921243619962880})}]
I20260927 11:21:56.124368 18452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:56.124465 18452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:56.125273 18452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9bd881691814acdbe165fbec476fbbb: Sending DeleteTablet for 3 replicas of tablet 779d6fdff4f4428f825b05c84c9046e5
I20260927 11:21:56.125574 18578 tablet_service.cc:1514] Processing DeleteTablet for tablet 779d6fdff4f4428f825b05c84c9046e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60636
I20260927 11:21:56.125626 18846 tablet_service.cc:1514] Processing DeleteTablet for tablet 779d6fdff4f4428f825b05c84c9046e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38640
I20260927 11:21:56.125648 18712 tablet_service.cc:1514] Processing DeleteTablet for tablet 779d6fdff4f4428f825b05c84c9046e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46672
I20260927 11:21:56.125833 18952 tablet_replica.cc:330] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251: stopping tablet replica
I20260927 11:21:56.125957 18952 raft_consensus.cc:2229] T 779d6fdff4f4428f825b05c84c9046e5 P da2dcebc900e469fb7d8337a8ab97251 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:21:56.126084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 18515
I20260927 11:21:56.126754 18954 tablet_replica.cc:330] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc: stopping tablet replica
I20260927 11:21:56.126852 18954 raft_consensus.cc:2229] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:21:56.126953 18954 raft_consensus.cc:2258] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:56.127203 18953 tablet_replica.cc:330] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad: stopping tablet replica
I20260927 11:21:56.127307 18953 raft_consensus.cc:2229] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:21:56.127414 18953 raft_consensus.cc:2258] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:56.127422 18954 ts_tablet_manager.cc:1905] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:56.127784 18953 ts_tablet_manager.cc:1905] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:56.129357 18953 ts_tablet_manager.cc:1918] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:56.129431 18953 log.cc:1196] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/779d6fdff4f4428f825b05c84c9046e5
I20260927 11:21:56.129572 18954 ts_tablet_manager.cc:1918] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:56.129650 18954 log.cc:1196] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/779d6fdff4f4428f825b05c84c9046e5
I20260927 11:21:56.129768 18953 ts_tablet_manager.cc:1939] T 779d6fdff4f4428f825b05c84c9046e5 P 40e79c73db8f471e9fc1575ff5fd4dad: Deleting consensus metadata
I20260927 11:21:56.129992 18954 ts_tablet_manager.cc:1939] T 779d6fdff4f4428f825b05c84c9046e5 P 0a53aba59dda4fe78d9f09e4b5c7babc: Deleting consensus metadata
I20260927 11:21:56.130182 18439 catalog_manager.cc:4841] TS 40e79c73db8f471e9fc1575ff5fd4dad (127.0.98.3:34275): tablet 779d6fdff4f4428f825b05c84c9046e5 (table test-table [id=c124dbec80df4095b75a8440767e367d]) successfully deleted
I20260927 11:21:56.130620 18440 catalog_manager.cc:4841] TS 0a53aba59dda4fe78d9f09e4b5c7babc (127.0.98.2:45365): tablet 779d6fdff4f4428f825b05c84c9046e5 (table test-table [id=c124dbec80df4095b75a8440767e367d]) successfully deleted
I20260927 11:21:56.134071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 18649
W20260927 11:21:56.134073 18439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41819 (error 108)
I20260927 11:21:56.134765 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.134620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41819 (local address 127.0.0.1:60666)
0927 11:21:56.134673 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:56.134723 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 11:21:56.134934 18439 catalog_manager.cc:4584] TS da2dcebc900e469fb7d8337a8ab97251 (127.0.98.1:41819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 779d6fdff4f4428f825b05c84c9046e5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41819: connect: Connection refused (error 111)
I20260927 11:21:56.139425 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 18783
I20260927 11:21:56.144419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 18421
2026-09-27T11:21:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1996 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T11:21:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:21:56Z Disabled control of system clock
I20260927 11:21:56.172389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45493
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:21:56.251933 18959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:56.252202 18959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:56.252238 18959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:56.253399 18959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:21:56.253480 18959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:56.253505 18959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:56.253538 18959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:21:56.253559 18959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:21:56.255002 18959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45493
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:56.255268 18959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:56.255555 18959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:56.260334 18965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:56.260322 18967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:56.260334 18964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:56.260558 18959 server_base.cc:1034] running on GCE node
I20260927 11:21:56.260774 18959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:56.261029 18959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:56.262212 18959 hybrid_clock.cc:648] HybridClock initialized: now 1790508116262183 us; error 50 us; skew 500 ppm
I20260927 11:21:56.263368 18959 webserver.cc:460] Webserver started at http://127.0.98.62:45097/ using document root <none> and password file <none>
I20260927 11:21:56.263595 18959 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:56.263638 18959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:56.263728 18959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:56.264801 18959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "6b1ea9d4b5ff440d8f0295a7c4ee8ca7"
format_stamp: "Formatted at 2026-09-27 11:21:56 on dist-test-slave-lsvg"
server_key: "1fcc17de91f7d74203752a67ee070152"
server_key_iv: "1144b5502b47a229ffdcfcb5cc5cba2e"
server_key_version: "encryptionkey@0"
I20260927 11:21:56.265172 18959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b1ea9d4b5ff440d8f0295a7c4ee8ca7"
format_stamp: "Formatted at 2026-09-27 11:21:56 on dist-test-slave-lsvg"
server_key: "1fcc17de91f7d74203752a67ee070152"
server_key_iv: "1144b5502b47a229ffdcfcb5cc5cba2e"
server_key_version: "encryptionkey@0"
I20260927 11:21:56.266520 18959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:56.267179 18973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:56.267367 18959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:21:56.267447 18959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "6b1ea9d4b5ff440d8f0295a7c4ee8ca7"
format_stamp: "Formatted at 2026-09-27 11:21:56 on dist-test-slave-lsvg"
server_key: "1fcc17de91f7d74203752a67ee070152"
server_key_iv: "1144b5502b47a229ffdcfcb5cc5cba2e"
server_key_version: "encryptionkey@0"
I20260927 11:21:56.267500 18959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:56.282166 18959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:56.282526 18959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:56.282663 18959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:56.287067 18959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45493
I20260927 11:21:56.287096 19025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45493 every 8 connection(s)
I20260927 11:21:56.287492 18959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:21:56.288455 19026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:56.290835 19026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7: Bootstrap starting.
I20260927 11:21:56.291406 19026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:56.291700 19026 log.cc:824] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:56.292356 19026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7: No bootstrap required, opened a new log
I20260927 11:21:56.293942 19026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1ea9d4b5ff440d8f0295a7c4ee8ca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260927 11:21:56.294111 19026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:56.294160 19026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1ea9d4b5ff440d8f0295a7c4ee8ca7, State: Initialized, Role: FOLLOWER
I20260927 11:21:56.294342 19026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1ea9d4b5ff440d8f0295a7c4ee8ca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260927 11:21:56.294440 19026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:21:56.294482 19026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:21:56.294529 19026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:56.295032 19026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1ea9d4b5ff440d8f0295a7c4ee8ca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260927 11:21:56.295147 19026 leader_election.cc:302] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b1ea9d4b5ff440d8f0295a7c4ee8ca7; no voters:
I20260927 11:21:56.295329 19026 leader_election.cc:288] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:21:56.295382 19029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:21:56.295572 19026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:21:56.295594 19029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [term 1 LEADER]: Becoming Leader. State: Replica: 6b1ea9d4b5ff440d8f0295a7c4ee8ca7, State: Running, Role: LEADER
I20260927 11:21:56.295686 19029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1ea9d4b5ff440d8f0295a7c4ee8ca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260927 11:21:56.296047 19031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b1ea9d4b5ff440d8f0295a7c4ee8ca7. Latest consensus state: current_term: 1 leader_uuid: "6b1ea9d4b5ff440d8f0295a7c4ee8ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1ea9d4b5ff440d8f0295a7c4ee8ca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } } }
I20260927 11:21:56.296136 19030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b1ea9d4b5ff440d8f0295a7c4ee8ca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1ea9d4b5ff440d8f0295a7c4ee8ca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } } }
I20260927 11:21:56.296154 19031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [sys.catalog]: This master's current role is: LEADER
I20260927 11:21:56.296255 19030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7 [sys.catalog]: This master's current role is: LEADER
I20260927 11:21:56.296653 19038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:21:56.296782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 18959
I20260927 11:21:56.296900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:21:56.297163 392 external_mini_cluster.cc:1428] Setting key 35e63df4bbddfd68295f004dc42d2b78
I20260927 11:21:56.297307 19038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:21:56.299134 19038 catalog_manager.cc:1347] Generated new cluster ID: 1d3e4dcbdfbe4336890fac0d76c7785e
I20260927 11:21:56.299194 19038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:21:56.303440 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.297614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.0.98.62:45493)
0927 11:21:56.298080 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.298089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.298154 (+ 65us) server_negotiation.cc:407] Connection header received
0927 11:21:56.298276 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.298282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.298337 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.298487 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:56.298732 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.300060 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.301176 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.301844 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.301969 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.302713 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:56.302717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:56.302717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:56.302931 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:56.302973 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:56.302990 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:56.303061 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.303128 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.303212 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":85,"threads_started":1}
I20260927 11:21:56.320091 19038 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:21:56.320871 19038 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:21:56.332002 19038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7: Generated new TSK 0
I20260927 11:21:56.332230 19038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:21:56.334892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45493
--builtin_ntp_servers=127.0.98.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:21:56.416364 19050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:56.416579 19050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:56.416608 19050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:56.417840 19050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:56.417881 19050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:56.417906 19050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:21:56.419260 19050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:56.419492 19050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:56.419818 19050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:56.425824 19055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:56.425882 19056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:56.425882 19058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:56.426250 19050 server_base.cc:1034] running on GCE node
I20260927 11:21:56.426483 19050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:56.426744 19050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:56.427913 19050 hybrid_clock.cc:648] HybridClock initialized: now 1790508116427901 us; error 33 us; skew 500 ppm
I20260927 11:21:56.434228 19050 webserver.cc:460] Webserver started at http://127.0.98.1:38489/ using document root <none> and password file <none>
I20260927 11:21:56.434517 19050 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:56.434590 19050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:56.434736 19050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:56.435849 19050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "2280f22b113e4dad806acbb9efd16d6f"
format_stamp: "Formatted at 2026-09-27 11:21:56 on dist-test-slave-lsvg"
server_key: "1af5b4c091d69b8952103862f9dc755c"
server_key_iv: "766a45c2f50fc030f0c74328de807825"
server_key_version: "encryptionkey@0"
I20260927 11:21:56.436230 19050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2280f22b113e4dad806acbb9efd16d6f"
format_stamp: "Formatted at 2026-09-27 11:21:56 on dist-test-slave-lsvg"
server_key: "1af5b4c091d69b8952103862f9dc755c"
server_key_iv: "766a45c2f50fc030f0c74328de807825"
server_key_version: "encryptionkey@0"
I20260927 11:21:56.437649 19050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:21:56.438414 19064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:56.438622 19050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:21:56.438728 19050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "2280f22b113e4dad806acbb9efd16d6f"
format_stamp: "Formatted at 2026-09-27 11:21:56 on dist-test-slave-lsvg"
server_key: "1af5b4c091d69b8952103862f9dc755c"
server_key_iv: "766a45c2f50fc030f0c74328de807825"
server_key_version: "encryptionkey@0"
I20260927 11:21:56.438805 19050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:56.445935 19050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:56.446234 19050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:56.446383 19050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:56.446607 19050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:56.446928 19050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:56.446981 19050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:56.447022 19050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:56.447049 19050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:56.452792 19050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43807
I20260927 11:21:56.452864 19177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43807 every 8 connection(s)
I20260927 11:21:56.453253 19050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:21:56.458305 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.454016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42775 (local address 127.0.98.62:45493)
0927 11:21:56.454082 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.454085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.454322 (+ 237us) server_negotiation.cc:407] Connection header received
0927 11:21:56.454798 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.454799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.454812 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.454865 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:56.455501 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.455989 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.457037 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.457290 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.457345 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.458064 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:56.458065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:56.458065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:56.458069 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:56.458081 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:56.458085 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:56.458142 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.458224 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.458248 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:21:56.458519 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.454023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:42775)
0927 11:21:56.454201 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:56.454234 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:56.454715 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:56.454917 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:56.454920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:56.454927 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:56.455418 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.455422 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.456069 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.456069 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.456963 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.456965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.457368 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.457369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.457478 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.457943 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:21:56.457943 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:56.457970 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:21:56.457990 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:56.458155 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:56.458159 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:56.458161 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:21:56.458214 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 11:21:56.458288 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260927 11:21:56.459043 19178 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45493
I20260927 11:21:56.459127 19178 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:56.459317 19178 heartbeater.cc:511] Master 127.0.98.62:45493 requested a full tablet report, sending...
I20260927 11:21:56.459810 18990 ts_manager.cc:194] Registered new tserver with Master: 2280f22b113e4dad806acbb9efd16d6f (127.0.98.1:43807)
I20260927 11:21:56.460031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 19050
I20260927 11:21:56.460146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:21:56.460325 392 external_mini_cluster.cc:1428] Setting key 30df9eeabbfcb1a3783a1248d3f65f76
I20260927 11:21:56.460656 18990 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42775
I20260927 11:21:56.461539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45493
--builtin_ntp_servers=127.0.98.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:21:56.545694 19181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:56.545969 19181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:56.545997 19181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:56.547195 19181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:56.547240 19181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:56.547267 19181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:21:56.548683 19181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:56.548945 19181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:56.549273 19181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:56.554399 19187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:56.554380 19189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:56.554538 19181 server_base.cc:1034] running on GCE node
W20260927 11:21:56.554370 19186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:56.554829 19181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:56.555089 19181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:56.556254 19181 hybrid_clock.cc:648] HybridClock initialized: now 1790508116556244 us; error 43 us; skew 500 ppm
I20260927 11:21:56.557536 19181 webserver.cc:460] Webserver started at http://127.0.98.2:38369/ using document root <none> and password file <none>
I20260927 11:21:56.557791 19181 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:56.557861 19181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:56.558012 19181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:56.559083 19181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "290c420a4f314a21bbb25b28202872e0"
format_stamp: "Formatted at 2026-09-27 11:21:56 on dist-test-slave-lsvg"
server_key: "d1db81df32742581285e77065c8fcadb"
server_key_iv: "742f9cb4210352132959cba5e5b24f33"
server_key_version: "encryptionkey@0"
I20260927 11:21:56.559448 19181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "290c420a4f314a21bbb25b28202872e0"
format_stamp: "Formatted at 2026-09-27 11:21:56 on dist-test-slave-lsvg"
server_key: "d1db81df32742581285e77065c8fcadb"
server_key_iv: "742f9cb4210352132959cba5e5b24f33"
server_key_version: "encryptionkey@0"
I20260927 11:21:56.560817 19181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:21:56.561697 19195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:56.561898 19181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:21:56.561998 19181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "290c420a4f314a21bbb25b28202872e0"
format_stamp: "Formatted at 2026-09-27 11:21:56 on dist-test-slave-lsvg"
server_key: "d1db81df32742581285e77065c8fcadb"
server_key_iv: "742f9cb4210352132959cba5e5b24f33"
server_key_version: "encryptionkey@0"
I20260927 11:21:56.562083 19181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:56.577314 19181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:56.577634 19181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:56.577788 19181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:56.578015 19181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:56.578364 19181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:56.578420 19181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:56.578456 19181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:56.578483 19181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:56.584189 19181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38479
I20260927 11:21:56.584237 19308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38479 every 8 connection(s)
I20260927 11:21:56.584563 19181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:21:56.586526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 19181
I20260927 11:21:56.586618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:21:56.586817 392 external_mini_cluster.cc:1428] Setting key fbf1abf5185e0fab02745d2c76a5e0f1
I20260927 11:21:56.587777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45493
--builtin_ntp_servers=127.0.98.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 11:21:56.590436 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.585356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55579 (local address 127.0.98.62:45493)
0927 11:21:56.585454 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.585456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.585679 (+ 223us) server_negotiation.cc:407] Connection header received
0927 11:21:56.586264 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.586265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.586279 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.586351 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:56.587218 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.587739 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.588820 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.589039 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.589148 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.590114 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:56.590115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:56.590115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:56.590120 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:56.590136 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:56.590140 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:56.590225 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.590311 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.590342 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 11:21:56.590663 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.585335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:55579)
0927 11:21:56.585530 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:56.585580 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:56.586138 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:56.586455 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:56.586459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:56.586475 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:56.587112 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.587119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.587852 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.587853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.588713 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.588715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.589156 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.589157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.589312 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.589945 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:21:56.589946 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:56.589975 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:21:56.589998 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:56.590228 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:56.590235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:56.590238 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:21:56.590315 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 11:21:56.590421 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260927 11:21:56.591228 19309 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45493
I20260927 11:21:56.591341 19309 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:56.591559 19309 heartbeater.cc:511] Master 127.0.98.62:45493 requested a full tablet report, sending...
I20260927 11:21:56.592024 18989 ts_manager.cc:194] Registered new tserver with Master: 290c420a4f314a21bbb25b28202872e0 (127.0.98.2:38479)
I20260927 11:21:56.592583 18989 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55579
W20260927 11:21:56.671622 19312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:56.671836 19312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:56.671859 19312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:56.672952 19312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:56.672988 19312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:56.673015 19312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:21:56.674378 19312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:56.674579 19312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:56.674829 19312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:56.679776 19320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:56.679776 19317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:56.679910 19312 server_base.cc:1034] running on GCE node
W20260927 11:21:56.680003 19318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:56.680253 19312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:56.680441 19312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:56.681599 19312 hybrid_clock.cc:648] HybridClock initialized: now 1790508116681587 us; error 27 us; skew 500 ppm
I20260927 11:21:56.683765 19312 webserver.cc:460] Webserver started at http://127.0.98.3:35933/ using document root <none> and password file <none>
I20260927 11:21:56.684074 19312 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:56.684141 19312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:56.684504 19312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:56.686357 19312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "b6db18c9d7c943f895358d9c84c6428e"
format_stamp: "Formatted at 2026-09-27 11:21:56 on dist-test-slave-lsvg"
server_key: "7f5f6836a8ebf525f4f2a5a48576ef07"
server_key_iv: "8028a1c9afc90dc376e6cabb1301de52"
server_key_version: "encryptionkey@0"
I20260927 11:21:56.686838 19312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b6db18c9d7c943f895358d9c84c6428e"
format_stamp: "Formatted at 2026-09-27 11:21:56 on dist-test-slave-lsvg"
server_key: "7f5f6836a8ebf525f4f2a5a48576ef07"
server_key_iv: "8028a1c9afc90dc376e6cabb1301de52"
server_key_version: "encryptionkey@0"
I20260927 11:21:56.689231 19312 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 11:21:56.690456 19326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:56.690672 19312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:56.690783 19312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "b6db18c9d7c943f895358d9c84c6428e"
format_stamp: "Formatted at 2026-09-27 11:21:56 on dist-test-slave-lsvg"
server_key: "7f5f6836a8ebf525f4f2a5a48576ef07"
server_key_iv: "8028a1c9afc90dc376e6cabb1301de52"
server_key_version: "encryptionkey@0"
I20260927 11:21:56.690861 19312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:56.709295 19312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:56.709697 19312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:56.709864 19312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:56.710120 19312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:56.710596 19312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:56.710644 19312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:56.710678 19312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:56.710702 19312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:56.718402 19312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39893
I20260927 11:21:56.718420 19439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39893 every 8 connection(s)
I20260927 11:21:56.718897 19312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:21:56.723012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 19312
I20260927 11:21:56.723145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:21:56.723376 392 external_mini_cluster.cc:1428] Setting key 5575421c82c1df0fded88f8eaf5cc52d
I20260927 11:21:56.724543 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.719668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51227 (local address 127.0.98.62:45493)
0927 11:21:56.719731 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.719734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.719990 (+ 256us) server_negotiation.cc:407] Connection header received
0927 11:21:56.720523 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.720525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.720538 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.720586 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:56.721341 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.721847 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.722837 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.723057 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.723135 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.724181 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:56.724183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:56.724183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:56.724188 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:56.724203 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:56.724207 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:56.724234 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.724382 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.724430 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 11:21:56.724673 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.719671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:51227)
0927 11:21:56.719866 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:56.719900 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:56.720416 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:56.720658 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:56.720661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:56.720669 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:56.721240 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.721246 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.721937 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.721938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.722732 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.722734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.723203 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.723204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.723341 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.724010 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:21:56.724011 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:56.724053 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:21:56.724084 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:56.724277 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:56.724282 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:56.724285 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:21:56.724359 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 11:21:56.724470 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260927 11:21:56.725200 19440 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45493
I20260927 11:21:56.725311 19440 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:56.725495 19440 heartbeater.cc:511] Master 127.0.98.62:45493 requested a full tablet report, sending...
I20260927 11:21:56.725903 18989 ts_manager.cc:194] Registered new tserver with Master: b6db18c9d7c943f895358d9c84c6428e (127.0.98.3:39893)
I20260927 11:21:56.726284 18989 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51227
I20260927 11:21:56.735500 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:21:56.739849 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.736870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.0.98.62:45493)
0927 11:21:56.736967 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.736970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.737104 (+ 134us) server_negotiation.cc:407] Connection header received
0927 11:21:56.737196 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.737197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.737211 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.737264 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:56.737423 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.737908 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.738658 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.738840 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.738898 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.739541 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:56.739542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:56.739543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:56.739547 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:56.739564 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:56.739568 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:56.739653 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.739712 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.739721 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 11:21:56.741513 18989 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:21:56.743093 18989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:21:56.749635 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.745303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39893 (local address 127.0.0.1:50578)
0927 11:21:56.745705 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:56.745748 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:56.745815 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:56.746111 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:56.746115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:56.746118 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:56.746181 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.746183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.747465 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.747466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.748482 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.748485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.749307 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.749308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.749391 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.749406 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 11:21:56.749489 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 11:21:56.749543 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":125,"threads_started":1}
I20260927 11:21:56.749630 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.745724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50578 (local address 127.0.98.3:39893)
0927 11:21:56.745920 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.745924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.745933 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:21:56.745973 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.745975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.745991 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.746070 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:56.746247 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.747371 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.748572 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.749223 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.749296 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.749382 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.749509 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.749547 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260927 11:21:56.749912 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.745409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.0.98.2:38479)
0927 11:21:56.745615 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.745621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.746227 (+ 606us) server_negotiation.cc:407] Connection header received
0927 11:21:56.746307 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.746309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.746329 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.746400 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:56.746609 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.747305 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.748800 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.749456 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.749497 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.749574 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.749791 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.749826 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":77,"threads_started":1}
I20260927 11:21:56.750118 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.745580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43807 (local address 127.0.0.1:44562)
0927 11:21:56.746307 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:56.746315 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:56.746364 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:56.746557 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:56.746558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:56.746560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:56.746594 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.746595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.747764 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.747765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.748816 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.748818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.749845 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.749846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.749907 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.749913 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 11:21:56.749990 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 11:21:56.750040 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260927 11:21:56.750409 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.745682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.0.98.1:43807)
0927 11:21:56.745857 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.745861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.746404 (+ 543us) server_negotiation.cc:407] Connection header received
0927 11:21:56.746454 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.746456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.746471 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.746549 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:56.746651 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.747684 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.748894 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.749646 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.749714 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.749810 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.750069 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.750118 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
I20260927 11:21:56.750591 19374 tablet_service.cc:1467] Processing CreateTablet for tablet f3f25844b73545c4a69ff923a42e4d3f (DEFAULT_TABLE table=test-table [id=5deedd5b389946dc97dcf3f4a5ff0cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:56.750984 19374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f25844b73545c4a69ff923a42e4d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:56.751103 19112 tablet_service.cc:1467] Processing CreateTablet for tablet f3f25844b73545c4a69ff923a42e4d3f (DEFAULT_TABLE table=test-table [id=5deedd5b389946dc97dcf3f4a5ff0cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:56.751425 19112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f25844b73545c4a69ff923a42e4d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:56.753182 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.745308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38479 (local address 127.0.0.1:43598)
0927 11:21:56.746169 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:56.746179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:56.746229 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:56.746470 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:56.746471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:56.746473 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:56.746506 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.746507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.747384 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.747385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.748707 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.748709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.749631 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.749632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.749694 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.749698 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:21:56.750067 (+ 369us) client_negotiation.cc:240] Negotiation successful
0927 11:21:56.750113 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":34,"thread_start_us":125,"threads_started":1}
I20260927 11:21:56.753842 19243 tablet_service.cc:1467] Processing CreateTablet for tablet f3f25844b73545c4a69ff923a42e4d3f (DEFAULT_TABLE table=test-table [id=5deedd5b389946dc97dcf3f4a5ff0cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:56.754150 19243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f25844b73545c4a69ff923a42e4d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:56.754707 19454 tablet_bootstrap.cc:492] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f: Bootstrap starting.
I20260927 11:21:56.755479 19454 tablet_bootstrap.cc:654] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:56.755815 19454 log.cc:824] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:56.756183 19455 tablet_bootstrap.cc:492] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e: Bootstrap starting.
I20260927 11:21:56.756585 19454 tablet_bootstrap.cc:492] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f: No bootstrap required, opened a new log
I20260927 11:21:56.756673 19454 ts_tablet_manager.cc:1397] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:21:56.756801 19455 tablet_bootstrap.cc:654] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:56.757165 19455 log.cc:824] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:56.757500 19458 tablet_bootstrap.cc:492] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0: Bootstrap starting.
I20260927 11:21:56.758004 19458 tablet_bootstrap.cc:654] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:56.758172 19455 tablet_bootstrap.cc:492] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e: No bootstrap required, opened a new log
I20260927 11:21:56.758244 19455 ts_tablet_manager.cc:1397] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 11:21:56.758345 19458 log.cc:824] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:56.759690 19454 raft_consensus.cc:348] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290c420a4f314a21bbb25b28202872e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38479 } } peers { permanent_uuid: "b6db18c9d7c943f895358d9c84c6428e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39893 } } peers { permanent_uuid: "2280f22b113e4dad806acbb9efd16d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260927 11:21:56.759859 19454 raft_consensus.cc:374] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:56.759895 19454 raft_consensus.cc:729] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2280f22b113e4dad806acbb9efd16d6f, State: Initialized, Role: FOLLOWER
I20260927 11:21:56.760021 19454 consensus_queue.cc:260] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290c420a4f314a21bbb25b28202872e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38479 } } peers { permanent_uuid: "b6db18c9d7c943f895358d9c84c6428e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39893 } } peers { permanent_uuid: "2280f22b113e4dad806acbb9efd16d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260927 11:21:56.760300 19454 ts_tablet_manager.cc:1428] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 11:21:56.760340 19178 heartbeater.cc:503] Master 127.0.98.62:45493 was elected leader, sending a full tablet report...
I20260927 11:21:56.760763 19455 raft_consensus.cc:348] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290c420a4f314a21bbb25b28202872e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38479 } } peers { permanent_uuid: "b6db18c9d7c943f895358d9c84c6428e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39893 } } peers { permanent_uuid: "2280f22b113e4dad806acbb9efd16d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260927 11:21:56.760912 19455 raft_consensus.cc:374] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:56.760957 19455 raft_consensus.cc:729] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6db18c9d7c943f895358d9c84c6428e, State: Initialized, Role: FOLLOWER
I20260927 11:21:56.761121 19455 consensus_queue.cc:260] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290c420a4f314a21bbb25b28202872e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38479 } } peers { permanent_uuid: "b6db18c9d7c943f895358d9c84c6428e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39893 } } peers { permanent_uuid: "2280f22b113e4dad806acbb9efd16d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260927 11:21:56.761390 19455 ts_tablet_manager.cc:1428] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 11:21:56.761417 19440 heartbeater.cc:503] Master 127.0.98.62:45493 was elected leader, sending a full tablet report...
I20260927 11:21:56.761993 19458 tablet_bootstrap.cc:492] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0: No bootstrap required, opened a new log
I20260927 11:21:56.762061 19458 ts_tablet_manager.cc:1397] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 11:21:56.764108 19458 raft_consensus.cc:348] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290c420a4f314a21bbb25b28202872e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38479 } } peers { permanent_uuid: "b6db18c9d7c943f895358d9c84c6428e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39893 } } peers { permanent_uuid: "2280f22b113e4dad806acbb9efd16d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260927 11:21:56.764220 19458 raft_consensus.cc:374] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:56.764241 19458 raft_consensus.cc:729] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290c420a4f314a21bbb25b28202872e0, State: Initialized, Role: FOLLOWER
I20260927 11:21:56.764323 19458 consensus_queue.cc:260] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290c420a4f314a21bbb25b28202872e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38479 } } peers { permanent_uuid: "b6db18c9d7c943f895358d9c84c6428e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39893 } } peers { permanent_uuid: "2280f22b113e4dad806acbb9efd16d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260927 11:21:56.764510 19458 ts_tablet_manager.cc:1428] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:21:56.764618 19309 heartbeater.cc:503] Master 127.0.98.62:45493 was elected leader, sending a full tablet report...
I20260927 11:21:56.833827 19462 raft_consensus.cc:482] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:21:56.833966 19462 raft_consensus.cc:504] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290c420a4f314a21bbb25b28202872e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38479 } } peers { permanent_uuid: "b6db18c9d7c943f895358d9c84c6428e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39893 } } peers { permanent_uuid: "2280f22b113e4dad806acbb9efd16d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260927 11:21:56.834378 19462 leader_election.cc:288] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6db18c9d7c943f895358d9c84c6428e (127.0.98.3:39893), 2280f22b113e4dad806acbb9efd16d6f (127.0.98.1:43807)
W20260927 11:21:56.835052 19310 tablet.cc:2369] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:21:56.838824 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.834851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50163 (local address 127.0.98.1:43807)
0927 11:21:56.834899 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.834901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.835018 (+ 117us) server_negotiation.cc:407] Connection header received
0927 11:21:56.835151 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.835152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.835167 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.835226 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:56.835464 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.836137 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.837795 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.838391 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.838473 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.838495 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.838703 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.838750 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 11:21:56.838824 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.834744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54805 (local address 127.0.98.3:39893)
0927 11:21:56.834857 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.834859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.834871 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:21:56.834903 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.834904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.834917 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.835022 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:56.835238 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.836048 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.837794 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.838391 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.838473 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.838485 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.838699 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.838750 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 11:21:56.838912 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.834759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43807 (local address 127.0.98.2:50163)
0927 11:21:56.834943 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:56.834953 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:56.835018 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:56.835248 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:56.835249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:56.835250 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:56.835330 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.835332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.836235 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.836236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.837678 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.837679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.838489 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.838490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.838580 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.838587 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 11:21:56.838651 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 11:21:56.838706 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":95,"thread_start_us":102,"threads_started":1}
I20260927 11:21:56.838786 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.834628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39893 (local address 127.0.98.2:54805)
0927 11:21:56.834723 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:56.834738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:56.834814 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:56.835056 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:56.835057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:56.835059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:56.835141 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.835143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.836235 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.836236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.837676 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:56.837679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.838489 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.838490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:56.838580 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.838587 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 11:21:56.838651 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 11:21:56.838707 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":113}
I20260927 11:21:56.839296 19132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f25844b73545c4a69ff923a42e4d3f" candidate_uuid: "290c420a4f314a21bbb25b28202872e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2280f22b113e4dad806acbb9efd16d6f" is_pre_election: true
I20260927 11:21:56.839526 19132 raft_consensus.cc:2454] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290c420a4f314a21bbb25b28202872e0 in term 0.
I20260927 11:21:56.839485 19394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f25844b73545c4a69ff923a42e4d3f" candidate_uuid: "290c420a4f314a21bbb25b28202872e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6db18c9d7c943f895358d9c84c6428e" is_pre_election: true
I20260927 11:21:56.839648 19394 raft_consensus.cc:2454] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 290c420a4f314a21bbb25b28202872e0 in term 0.
I20260927 11:21:56.839805 19198 leader_election.cc:302] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2280f22b113e4dad806acbb9efd16d6f, 290c420a4f314a21bbb25b28202872e0; no voters:
I20260927 11:21:56.840008 19462 raft_consensus.cc:2781] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:21:56.840080 19462 raft_consensus.cc:482] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:21:56.840147 19462 raft_consensus.cc:3037] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:56.840924 19462 raft_consensus.cc:504] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290c420a4f314a21bbb25b28202872e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38479 } } peers { permanent_uuid: "b6db18c9d7c943f895358d9c84c6428e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39893 } } peers { permanent_uuid: "2280f22b113e4dad806acbb9efd16d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260927 11:21:56.841102 19462 leader_election.cc:288] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [CANDIDATE]: Term 1 election: Requested vote from peers b6db18c9d7c943f895358d9c84c6428e (127.0.98.3:39893), 2280f22b113e4dad806acbb9efd16d6f (127.0.98.1:43807)
I20260927 11:21:56.841231 19394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f25844b73545c4a69ff923a42e4d3f" candidate_uuid: "290c420a4f314a21bbb25b28202872e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6db18c9d7c943f895358d9c84c6428e"
I20260927 11:21:56.841284 19132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f25844b73545c4a69ff923a42e4d3f" candidate_uuid: "290c420a4f314a21bbb25b28202872e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2280f22b113e4dad806acbb9efd16d6f"
I20260927 11:21:56.841321 19394 raft_consensus.cc:3037] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:56.841353 19132 raft_consensus.cc:3037] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:56.842314 19394 raft_consensus.cc:2454] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290c420a4f314a21bbb25b28202872e0 in term 1.
I20260927 11:21:56.842314 19132 raft_consensus.cc:2454] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 290c420a4f314a21bbb25b28202872e0 in term 1.
I20260927 11:21:56.842466 19198 leader_election.cc:302] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 290c420a4f314a21bbb25b28202872e0, b6db18c9d7c943f895358d9c84c6428e; no voters:
I20260927 11:21:56.842578 19462 raft_consensus.cc:2781] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:21:56.842793 19462 raft_consensus.cc:686] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [term 1 LEADER]: Becoming Leader. State: Replica: 290c420a4f314a21bbb25b28202872e0, State: Running, Role: LEADER
I20260927 11:21:56.842923 19462 consensus_queue.cc:237] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290c420a4f314a21bbb25b28202872e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38479 } } peers { permanent_uuid: "b6db18c9d7c943f895358d9c84c6428e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39893 } } peers { permanent_uuid: "2280f22b113e4dad806acbb9efd16d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } }
I20260927 11:21:56.843745 18989 catalog_manager.cc:5495] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 290c420a4f314a21bbb25b28202872e0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "290c420a4f314a21bbb25b28202872e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "290c420a4f314a21bbb25b28202872e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6db18c9d7c943f895358d9c84c6428e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2280f22b113e4dad806acbb9efd16d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 } health_report { overall_health: UNKNOWN } } }
I20260927 11:21:56.867354 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.865106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43608 (local address 127.0.98.2:38479)
0927 11:21:56.865146 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:56.865148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:56.865160 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:21:56.865268 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:56.865269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:56.865288 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:56.865353 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:21:56.865485 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.865961 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.866679 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:56.866880 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:56.866952 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:56.867068 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:21:56.867155 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:21:56.867224 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:56.867270 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 11:21:56.867277 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:21:56.869314 19394 raft_consensus.cc:1261] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e [term 1 FOLLOWER]: Refusing update from remote peer 290c420a4f314a21bbb25b28202872e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:56.869330 19132 raft_consensus.cc:1261] T f3f25844b73545c4a69ff923a42e4d3f P 2280f22b113e4dad806acbb9efd16d6f [term 1 FOLLOWER]: Refusing update from remote peer 290c420a4f314a21bbb25b28202872e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:56.869777 19464 consensus_queue.cc:1035] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6db18c9d7c943f895358d9c84c6428e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:56.869932 19462 consensus_queue.cc:1035] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2280f22b113e4dad806acbb9efd16d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:56.872493 19467 mvcc.cc:205] Tried to move back new op lower bound from 7333921246693339136 to 7333921246589202432. Current Snapshot: MvccSnapshot[applied={T|T < 7333921246693339136}]
I20260927 11:21:56.872776 19469 mvcc.cc:205] Tried to move back new op lower bound from 7333921246693339136 to 7333921246589202432. Current Snapshot: MvccSnapshot[applied={T|T < 7333921246693339136}]
I20260927 11:21:56.873375 19468 mvcc.cc:205] Tried to move back new op lower bound from 7333921246693339136 to 7333921246589202432. Current Snapshot: MvccSnapshot[applied={T|T < 7333921246693339136}]
I20260927 11:21:56.874822 18990 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:56.874917 18990 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:56.875603 18990 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b1ea9d4b5ff440d8f0295a7c4ee8ca7: Sending DeleteTablet for 3 replicas of tablet f3f25844b73545c4a69ff923a42e4d3f
I20260927 11:21:56.875892 19243 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f25844b73545c4a69ff923a42e4d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:56 UTC) from {username='slave'} at 127.0.0.1:43598
I20260927 11:21:56.875952 19112 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f25844b73545c4a69ff923a42e4d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:56 UTC) from {username='slave'} at 127.0.0.1:44562
I20260927 11:21:56.876062 19374 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f25844b73545c4a69ff923a42e4d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:56 UTC) from {username='slave'} at 127.0.0.1:50578
I20260927 11:21:56.876242 19477 tablet_replica.cc:330] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0: stopping tablet replica
I20260927 11:21:56.876294 19479 tablet_replica.cc:330] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e: stopping tablet replica
I20260927 11:21:56.876394 19479 raft_consensus.cc:2229] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:21:56.876408 19477 raft_consensus.cc:2229] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:21:56.876513 19479 raft_consensus.cc:2258] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:56.876719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 19050
I20260927 11:21:56.876746 19477 raft_consensus.cc:2258] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:56.879274 19477 ts_tablet_manager.cc:1905] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:56.879487 19479 ts_tablet_manager.cc:1905] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:56.881207 19477 ts_tablet_manager.cc:1918] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:56.881213 19479 ts_tablet_manager.cc:1918] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:56.881292 19479 log.cc:1196] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/f3f25844b73545c4a69ff923a42e4d3f
I20260927 11:21:56.881291 19477 log.cc:1196] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/f3f25844b73545c4a69ff923a42e4d3f
I20260927 11:21:56.881573 19477 ts_tablet_manager.cc:1939] T f3f25844b73545c4a69ff923a42e4d3f P 290c420a4f314a21bbb25b28202872e0: Deleting consensus metadata
I20260927 11:21:56.881623 19479 ts_tablet_manager.cc:1939] T f3f25844b73545c4a69ff923a42e4d3f P b6db18c9d7c943f895358d9c84c6428e: Deleting consensus metadata
I20260927 11:21:56.881991 18977 catalog_manager.cc:4841] TS 290c420a4f314a21bbb25b28202872e0 (127.0.98.2:38479): tablet f3f25844b73545c4a69ff923a42e4d3f (table test-table [id=5deedd5b389946dc97dcf3f4a5ff0cb4]) successfully deleted
I20260927 11:21:56.882035 18976 catalog_manager.cc:4841] TS b6db18c9d7c943f895358d9c84c6428e (127.0.98.3:39893): tablet f3f25844b73545c4a69ff923a42e4d3f (table test-table [id=5deedd5b389946dc97dcf3f4a5ff0cb4]) successfully deleted
W20260927 11:21:56.884609 18976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43807 (error 108)
I20260927 11:21:56.884795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 19181
I20260927 11:21:56.886530 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:56.886381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43807 (local address 127.0.0.1:44576)
0927 11:21:56.886441 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:56.886487 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 11:21:56.886660 18976 catalog_manager.cc:4584] TS 2280f22b113e4dad806acbb9efd16d6f (127.0.98.1:43807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3f25844b73545c4a69ff923a42e4d3f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43807: connect: Connection refused (error 111)
I20260927 11:21:56.889230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 19312
I20260927 11:21:56.893426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 18959
2026-09-27T11:21:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (747 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:21:56 dist-test-slave-lsvg krb5kdc[19483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:21:56 dist-test-slave-lsvg krb5kdc[19483](info): set up 2 sockets
Sep 27 11:21:56 dist-test-slave-lsvg krb5kdc[19483](info): commencing operation
krb5kdc: starting...
W20260927 11:21:57.928881 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:21:58 dist-test-slave-lsvg krb5kdc[19483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:21:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:21:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:21:58.048950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35755
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:58.132396 19496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:58.132627 19496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:58.132651 19496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:58.133991 19496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:21:58.134037 19496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:58.134052 19496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:58.134064 19496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:21:58.134078 19496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:21:58.135591 19496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:58.135844 19496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:58.136132 19496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:58.141279 19502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:58.141268 19504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:58.141268 19501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:58.141362 19496 server_base.cc:1034] running on GCE node
I20260927 11:21:58.141884 19496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:58.142222 19496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:58.143440 19496 hybrid_clock.cc:648] HybridClock initialized: now 1790508118143421 us; error 40 us; skew 500 ppm
Sep 27 11:21:58 dist-test-slave-lsvg krb5kdc[19483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:58.146202 19496 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:21:58.146754 19496 webserver.cc:460] Webserver started at http://127.0.98.62:41375/ using document root <none> and password file <none>
I20260927 11:21:58.147006 19496 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:58.147078 19496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:58.147222 19496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:58.148309 19496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "5d6241aa8d8a442288200b3743beef4a"
format_stamp: "Formatted at 2026-09-27 11:21:58 on dist-test-slave-lsvg"
server_key: "d76cda6371efbd1e04a0281e6d4e0550"
server_key_iv: "cdb506d710d074834c95abd479fa5bff"
server_key_version: "encryptionkey@0"
I20260927 11:21:58.148687 19496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d6241aa8d8a442288200b3743beef4a"
format_stamp: "Formatted at 2026-09-27 11:21:58 on dist-test-slave-lsvg"
server_key: "d76cda6371efbd1e04a0281e6d4e0550"
server_key_iv: "cdb506d710d074834c95abd479fa5bff"
server_key_version: "encryptionkey@0"
I20260927 11:21:58.150118 19496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:21:58.150960 19511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:58.151142 19496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:58.151254 19496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "5d6241aa8d8a442288200b3743beef4a"
format_stamp: "Formatted at 2026-09-27 11:21:58 on dist-test-slave-lsvg"
server_key: "d76cda6371efbd1e04a0281e6d4e0550"
server_key_iv: "cdb506d710d074834c95abd479fa5bff"
server_key_version: "encryptionkey@0"
I20260927 11:21:58.151333 19496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:58.165604 19496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:58.167680 19496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:58.167845 19496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:58.171880 19496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35755
I20260927 11:21:58.171919 19563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35755 every 8 connection(s)
I20260927 11:21:58.172292 19496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:21:58.173251 19564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:58.174574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 19496
I20260927 11:21:58.174669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:21:58.174896 392 external_mini_cluster.cc:1428] Setting key fd46f0495bc597342e8a023447642f7a
I20260927 11:21:58.176153 19564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a: Bootstrap starting.
I20260927 11:21:58.176888 19564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:58.177222 19564 log.cc:824] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:58.177983 19564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a: No bootstrap required, opened a new log
I20260927 11:21:58.180238 19564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6241aa8d8a442288200b3743beef4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35755 } }
I20260927 11:21:58.180423 19564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:58.180469 19564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d6241aa8d8a442288200b3743beef4a, State: Initialized, Role: FOLLOWER
I20260927 11:21:58.180619 19564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6241aa8d8a442288200b3743beef4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35755 } }
I20260927 11:21:58.180709 19564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:21:58.180752 19564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:21:58.180806 19564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:58.181450 19564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6241aa8d8a442288200b3743beef4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35755 } }
I20260927 11:21:58.181572 19564 leader_election.cc:302] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d6241aa8d8a442288200b3743beef4a; no voters:
I20260927 11:21:58.181795 19564 leader_election.cc:288] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:21:58.181821 19569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:21:58.181937 19569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [term 1 LEADER]: Becoming Leader. State: Replica: 5d6241aa8d8a442288200b3743beef4a, State: Running, Role: LEADER
I20260927 11:21:58.182034 19569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6241aa8d8a442288200b3743beef4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35755 } }
I20260927 11:21:58.182080 19564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:21:58.182343 19569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d6241aa8d8a442288200b3743beef4a. Latest consensus state: current_term: 1 leader_uuid: "5d6241aa8d8a442288200b3743beef4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6241aa8d8a442288200b3743beef4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35755 } } }
I20260927 11:21:58.182422 19569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [sys.catalog]: This master's current role is: LEADER
I20260927 11:21:58.182354 19570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d6241aa8d8a442288200b3743beef4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6241aa8d8a442288200b3743beef4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35755 } } }
I20260927 11:21:58.182621 19570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a [sys.catalog]: This master's current role is: LEADER
Sep 27 11:21:58 dist-test-slave-lsvg krb5kdc[19483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:21:58.183410 19573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:21:58.183928 19573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:21:58.185729 19573 catalog_manager.cc:1347] Generated new cluster ID: e05db8ede1924b89a654ecf468ae4073
I20260927 11:21:58.185781 19573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:21:58.187110 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.175506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38688 (local address 127.0.98.62:35755)
0927 11:21:58.175715 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:58.175726 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:58.175763 (+ 37us) server_negotiation.cc:407] Connection header received
0927 11:21:58.176320 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:58.176329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:58.176390 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:58.176557 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:58.177437 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.178807 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.179554 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.180258 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.180371 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.183305 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:58.183309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:58.183310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:58.183322 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:58.185444 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:58.185873 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:58.185876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:58.185876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:58.185977 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:58.186166 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:58.186166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:58.186166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:58.186445 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:58.186556 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:58.186774 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 11:21:58.186904 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260927 11:21:58.190337 19573 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:21:58.191022 19573 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:21:58.197439 19573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a: Generated new TSK 0
I20260927 11:21:58.197633 19573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:21:58.227777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35755
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:58.310539 19590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:58.310813 19590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:58.310859 19590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:58.312183 19590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:58.312251 19590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:58.312299 19590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:21:58.314016 19590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:58.314318 19590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:58.314569 19590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:58.319927 19598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:58.319936 19595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:58.319932 19596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:58.320191 19590 server_base.cc:1034] running on GCE node
I20260927 11:21:58.320437 19590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:58.320708 19590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:58.321897 19590 hybrid_clock.cc:648] HybridClock initialized: now 1790508118321879 us; error 47 us; skew 500 ppm
Sep 27 11:21:58 dist-test-slave-lsvg krb5kdc[19483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:58.324563 19590 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:21:58.325042 19590 webserver.cc:460] Webserver started at http://127.0.98.1:45041/ using document root <none> and password file <none>
I20260927 11:21:58.325348 19590 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:58.325419 19590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:58.325572 19590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:58.326695 19590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "c1fe910373ac4c0fb642655949f72cbd"
format_stamp: "Formatted at 2026-09-27 11:21:58 on dist-test-slave-lsvg"
server_key: "5283f089b52ea67d5b20ade1140f2f95"
server_key_iv: "d3fe178b122a000d8e40b3f27e517c9d"
server_key_version: "encryptionkey@0"
I20260927 11:21:58.327103 19590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c1fe910373ac4c0fb642655949f72cbd"
format_stamp: "Formatted at 2026-09-27 11:21:58 on dist-test-slave-lsvg"
server_key: "5283f089b52ea67d5b20ade1140f2f95"
server_key_iv: "d3fe178b122a000d8e40b3f27e517c9d"
server_key_version: "encryptionkey@0"
I20260927 11:21:58.328539 19590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:21:58.329408 19605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:58.329600 19590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:58.329708 19590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "c1fe910373ac4c0fb642655949f72cbd"
format_stamp: "Formatted at 2026-09-27 11:21:58 on dist-test-slave-lsvg"
server_key: "5283f089b52ea67d5b20ade1140f2f95"
server_key_iv: "d3fe178b122a000d8e40b3f27e517c9d"
server_key_version: "encryptionkey@0"
I20260927 11:21:58.329788 19590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:58.352042 19590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:58.354496 19590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:58.354642 19590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:58.354889 19590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:58.355254 19590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:58.355317 19590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:58.355353 19590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:58.355388 19590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:58.361107 19590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41861
I20260927 11:21:58.361176 19718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41861 every 8 connection(s)
I20260927 11:21:58.361531 19590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:21:58.363215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 19590
I20260927 11:21:58.363431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:21:58.363703 392 external_mini_cluster.cc:1428] Setting key 78a9daa39f048c57710a87cb3e2505bf
Sep 27 11:21:58 dist-test-slave-lsvg krb5kdc[19483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 11:21:58.372135 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.362262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55015 (local address 127.0.98.62:35755)
0927 11:21:58.362322 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:58.362324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:58.362539 (+ 215us) server_negotiation.cc:407] Connection header received
0927 11:21:58.363600 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:58.363602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:58.363616 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:58.363710 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:58.364379 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.364889 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.366368 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.366598 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.366655 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.369184 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:58.369186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:58.369186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:58.369194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:58.370651 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:58.371100 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:58.371101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:58.371102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:58.371165 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:58.371428 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:58.371429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:58.371429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:58.371575 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:58.371653 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:58.371945 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 11:21:58.372037 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:21:58.372364 19721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.362240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35755 (local address 127.0.98.1:55015)
0927 11:21:58.362420 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:58.362456 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:58.363460 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:58.363792 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:58.363797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:58.364088 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:58.364281 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.364285 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.365006 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.365007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.366268 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.366271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.366682 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.366684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.366831 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.367381 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:58.367393 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:58.369008 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:58.370753 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:58.370762 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:58.370765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:58.371003 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:58.371235 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:58.371237 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:58.371237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:58.371350 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:58.371644 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:58.371650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:58.371801 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 11:21:58.371910 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 11:21:58.372101 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260927 11:21:58.372881 19719 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35755
I20260927 11:21:58.372982 19719 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:58.373211 19719 heartbeater.cc:511] Master 127.0.98.62:35755 requested a full tablet report, sending...
I20260927 11:21:58.373813 19528 ts_manager.cc:194] Registered new tserver with Master: c1fe910373ac4c0fb642655949f72cbd (127.0.98.1:41861)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 11:21:58.374567 19528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:21:58.390345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35755
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:58.474522 19724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:58.474823 19724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:58.474851 19724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:58.476055 19724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:58.476104 19724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:58.476131 19724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:21:58.477710 19724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:58.477968 19724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:58.478235 19724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:58.483469 19729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:58.483526 19730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:58.483675 19724 server_base.cc:1034] running on GCE node
W20260927 11:21:58.483475 19732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:58.483994 19724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:58.484251 19724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:58.485436 19724 hybrid_clock.cc:648] HybridClock initialized: now 1790508118485428 us; error 52 us; skew 500 ppm
Sep 27 11:21:58 dist-test-slave-lsvg krb5kdc[19483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:58.487926 19724 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:21:58.488317 19724 webserver.cc:460] Webserver started at http://127.0.98.2:40963/ using document root <none> and password file <none>
I20260927 11:21:58.488549 19724 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:58.488596 19724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:58.488730 19724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:58.489840 19724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "8980daae73a24c80bc90a56d59a8f565"
format_stamp: "Formatted at 2026-09-27 11:21:58 on dist-test-slave-lsvg"
server_key: "c20ecf4c6c4a50059dd2a46afaaca37e"
server_key_iv: "ea5e648ee7ca4261aae8079ad1dba39b"
server_key_version: "encryptionkey@0"
I20260927 11:21:58.490245 19724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8980daae73a24c80bc90a56d59a8f565"
format_stamp: "Formatted at 2026-09-27 11:21:58 on dist-test-slave-lsvg"
server_key: "c20ecf4c6c4a50059dd2a46afaaca37e"
server_key_iv: "ea5e648ee7ca4261aae8079ad1dba39b"
server_key_version: "encryptionkey@0"
I20260927 11:21:58.491544 19724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:21:58.492259 19739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:58.492452 19724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:21:58.492535 19724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "8980daae73a24c80bc90a56d59a8f565"
format_stamp: "Formatted at 2026-09-27 11:21:58 on dist-test-slave-lsvg"
server_key: "c20ecf4c6c4a50059dd2a46afaaca37e"
server_key_iv: "ea5e648ee7ca4261aae8079ad1dba39b"
server_key_version: "encryptionkey@0"
I20260927 11:21:58.492606 19724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:58.508360 19724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:58.510756 19724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:58.510871 19724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:58.511071 19724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:58.511335 19724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:58.511366 19724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:58.511389 19724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:58.511402 19724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:58.517033 19724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43031
I20260927 11:21:58.517127 19852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43031 every 8 connection(s)
I20260927 11:21:58.517467 19724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:21:58 dist-test-slave-lsvg krb5kdc[19483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:21:58.525514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 19724
I20260927 11:21:58.525610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:21:58.525823 392 external_mini_cluster.cc:1428] Setting key e824e56646607a2fb7f88e40d0868954
I20260927 11:21:58.527486 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.518226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44997 (local address 127.0.98.62:35755)
0927 11:21:58.518300 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:58.518301 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:58.518542 (+ 241us) server_negotiation.cc:407] Connection header received
0927 11:21:58.519533 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:58.519535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:58.519547 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:58.519605 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:58.520250 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.520747 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.521832 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.522034 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.522139 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.524376 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:58.524377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:58.524378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:58.524392 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:58.525895 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:58.526365 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:58.526366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:58.526367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:58.526423 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:58.526756 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:58.526757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:58.526758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:58.526918 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:58.526988 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:58.527251 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 11:21:58.527347 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 11:21:58.527496 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.518220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35755 (local address 127.0.98.2:44997)
0927 11:21:58.518401 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:58.518437 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:58.519403 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:58.519694 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:58.519698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:58.519951 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:58.520147 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.520151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.520856 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.520857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.521724 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.521727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.522150 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.522151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.522279 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.522810 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:58.522820 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:58.524267 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:58.525997 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:58.526005 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:58.526008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:58.526291 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:58.526524 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:58.526525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:58.526526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:58.526675 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:58.526985 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:58.526990 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:58.527119 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 11:21:58.527204 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 11:21:58.527330 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260927 11:21:58.528218 19853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35755
I20260927 11:21:58.528316 19853 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:58.528512 19853 heartbeater.cc:511] Master 127.0.98.62:35755 requested a full tablet report, sending...
I20260927 11:21:58.528920 19528 ts_manager.cc:194] Registered new tserver with Master: 8980daae73a24c80bc90a56d59a8f565 (127.0.98.2:43031)
I20260927 11:21:58.529485 19528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44997
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:21:58.552395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35755
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:21:58.633868 19858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:58.634089 19858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:58.634114 19858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:58.635308 19858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:58.635352 19858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:58.635380 19858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:21:58.636986 19858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:58.637271 19858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:58.637511 19858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:58.642663 19864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:58.642628 19863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:58.642724 19866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:58.642756 19858 server_base.cc:1034] running on GCE node
I20260927 11:21:58.643143 19858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:58.643399 19858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:58.644549 19858 hybrid_clock.cc:648] HybridClock initialized: now 1790508118644519 us; error 55 us; skew 500 ppm
Sep 27 11:21:58 dist-test-slave-lsvg krb5kdc[19483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:21:58.647195 19858 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:21:58.647683 19858 webserver.cc:460] Webserver started at http://127.0.98.3:37195/ using document root <none> and password file <none>
I20260927 11:21:58.647948 19858 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:58.647998 19858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:58.648169 19858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:58.649317 19858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "253695c10c8942bfbf721550a491605a"
format_stamp: "Formatted at 2026-09-27 11:21:58 on dist-test-slave-lsvg"
server_key: "7670a940276dbe6659a80061b49789a4"
server_key_iv: "cfa06f28e206806adaf3870e1978e4c3"
server_key_version: "encryptionkey@0"
I20260927 11:21:58.649679 19858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "253695c10c8942bfbf721550a491605a"
format_stamp: "Formatted at 2026-09-27 11:21:58 on dist-test-slave-lsvg"
server_key: "7670a940276dbe6659a80061b49789a4"
server_key_iv: "cfa06f28e206806adaf3870e1978e4c3"
server_key_version: "encryptionkey@0"
I20260927 11:21:58.651075 19858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:21:58.651813 19873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:58.652009 19858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:21:58.652081 19858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "253695c10c8942bfbf721550a491605a"
format_stamp: "Formatted at 2026-09-27 11:21:58 on dist-test-slave-lsvg"
server_key: "7670a940276dbe6659a80061b49789a4"
server_key_iv: "cfa06f28e206806adaf3870e1978e4c3"
server_key_version: "encryptionkey@0"
I20260927 11:21:58.652173 19858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:58.662626 19858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:58.664690 19858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:58.664841 19858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:58.665167 19858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:58.665493 19858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:58.665526 19858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:58.665548 19858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:58.665592 19858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:58.671913 19858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40853
I20260927 11:21:58.671948 19986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40853 every 8 connection(s)
I20260927 11:21:58.672288 19858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:21:58.677304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 19858
I20260927 11:21:58.677424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:21:58.677616 392 external_mini_cluster.cc:1428] Setting key 5c5a836a0d47944c73822a4b9ebda38e
Sep 27 11:21:58 dist-test-slave-lsvg krb5kdc[19483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:21:58.681814 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.673002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55729 (local address 127.0.98.62:35755)
0927 11:21:58.673118 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:58.673120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:58.673345 (+ 225us) server_negotiation.cc:407] Connection header received
0927 11:21:58.674287 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:58.674288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:58.674299 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:58.674336 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:58.674941 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.675427 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.676421 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.676622 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.676686 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.679348 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:58.679350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:58.679350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:58.679356 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:58.680570 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:58.680910 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:58.680911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:58.680911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:58.680968 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:58.681240 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:58.681241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:58.681241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:58.681342 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:58.681395 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:58.681670 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 11:21:58.681751 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 11:21:58.681946 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.672973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35755 (local address 127.0.98.3:55729)
0927 11:21:58.673218 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:58.673262 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:58.674186 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:58.674416 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:58.674419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:58.674679 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:58.674868 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.674873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.675504 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.675504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.676325 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.676327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.676722 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.676723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.676831 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.677530 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:21:58.677540 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:58.679265 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:58.680638 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:58.680644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:58.680645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:58.680847 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:58.681023 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:21:58.681024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:21:58.681024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:21:58.681183 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:21:58.681399 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:58.681403 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:58.681514 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 11:21:58.681590 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 11:21:58.681764 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":120,"threads_started":1}
I20260927 11:21:58.682438 19987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35755
I20260927 11:21:58.682520 19987 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:58.682744 19987 heartbeater.cc:511] Master 127.0.98.62:35755 requested a full tablet report, sending...
I20260927 11:21:58.683115 19528 ts_manager.cc:194] Registered new tserver with Master: 253695c10c8942bfbf721550a491605a (127.0.98.3:40853)
I20260927 11:21:58.683557 19528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55729
I20260927 11:21:58.689438 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:21:58.697142 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.690792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.0.98.62:35755)
0927 11:21:58.690842 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:58.690844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:58.690942 (+ 98us) server_negotiation.cc:407] Connection header received
0927 11:21:58.690990 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:58.690990 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:58.690998 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:58.691044 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:58.691581 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.692127 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.692845 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.693038 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.693174 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.694251 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:58.694252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:58.694253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:21:58.694259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:58.695663 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:58.696084 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:58.696085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:58.696085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:58.696135 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:21:58.696394 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:21:58.696395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:21:58.696395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:21:58.696557 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:58.696651 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:58.696846 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 11:21:58.696999 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 11:21:58.699029 19528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:21:58.700657 19528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:21:58.707434 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.703372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.0.98.1:41861)
0927 11:21:58.703721 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:58.703725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:58.703923 (+ 198us) server_negotiation.cc:407] Connection header received
0927 11:21:58.703985 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:58.703987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:58.704005 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:58.704088 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:58.704243 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.704955 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.706366 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.707017 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.707091 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.707148 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:58.707303 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 11:21:58.707344 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":286,"threads_started":1}
I20260927 11:21:58.707423 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.703032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41861 (local address 127.0.0.1:53062)
0927 11:21:58.703781 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:58.703791 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:58.703846 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:58.704080 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:58.704081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:58.704083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:58.704156 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.704158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.705086 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.705087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.706280 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.706282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.707105 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.707106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.707191 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.707206 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 11:21:58.707278 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 11:21:58.707332 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":6,"thread_start_us":74,"threads_started":1}
I20260927 11:21:58.707635 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.703193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.0.98.3:40853)
0927 11:21:58.703374 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:58.703416 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:58.703429 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:21:58.703472 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:58.703474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:58.703491 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:58.703580 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:58.704196 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.705359 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.706495 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.707130 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.707193 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.707247 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:58.707468 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 11:21:58.707561 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":115,"threads_started":1}
I20260927 11:21:58.707795 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.703031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40853 (local address 127.0.0.1:35368)
0927 11:21:58.703166 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:58.703209 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:58.703272 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:58.703684 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:58.703688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:58.703690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:58.703786 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.703788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.705478 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.705479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.706409 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.706411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.707280 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.707281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.707334 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.707336 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:21:58.707381 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 11:21:58.707412 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":46,"threads_started":1}
I20260927 11:21:58.708122 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.703223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43031 (local address 127.0.0.1:49126)
0927 11:21:58.703910 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:58.703918 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:58.703973 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:58.704171 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:58.704172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:58.704173 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:58.704210 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.704211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.705484 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.705485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.706733 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.706735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.707862 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.707864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.707945 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.707949 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:21:58.708001 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 11:21:58.708038 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":27,"thread_start_us":62,"threads_started":1}
I20260927 11:21:58.708357 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.703445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.0.98.2:43031)
0927 11:21:58.703616 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:58.703621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:58.704029 (+ 408us) server_negotiation.cc:407] Connection header received
0927 11:21:58.704072 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:58.704074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:58.704087 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:58.704159 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:58.704267 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.705359 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.706820 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.707707 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.707778 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.707828 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:58.708113 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 11:21:58.708301 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260927 11:21:58.708379 19921 tablet_service.cc:1467] Processing CreateTablet for tablet c0745346f8cf4d55876038d5f4f9fff1 (DEFAULT_TABLE table=test-table [id=6662aa4666c942a3818dbe85584fde18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:58.708380 19653 tablet_service.cc:1467] Processing CreateTablet for tablet c0745346f8cf4d55876038d5f4f9fff1 (DEFAULT_TABLE table=test-table [id=6662aa4666c942a3818dbe85584fde18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:58.708731 19653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0745346f8cf4d55876038d5f4f9fff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:58.708731 19921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0745346f8cf4d55876038d5f4f9fff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:58.709693 19787 tablet_service.cc:1467] Processing CreateTablet for tablet c0745346f8cf4d55876038d5f4f9fff1 (DEFAULT_TABLE table=test-table [id=6662aa4666c942a3818dbe85584fde18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:58.710003 19787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0745346f8cf4d55876038d5f4f9fff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:58.712052 20004 tablet_bootstrap.cc:492] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a: Bootstrap starting.
I20260927 11:21:58.712587 20006 tablet_bootstrap.cc:492] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565: Bootstrap starting.
I20260927 11:21:58.712846 20004 tablet_bootstrap.cc:654] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:58.713225 20006 tablet_bootstrap.cc:654] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:58.713253 20005 tablet_bootstrap.cc:492] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd: Bootstrap starting.
I20260927 11:21:58.713510 20004 log.cc:824] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:58.713793 20006 log.cc:824] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:58.713893 20005 tablet_bootstrap.cc:654] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:58.714232 20004 tablet_bootstrap.cc:492] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a: No bootstrap required, opened a new log
I20260927 11:21:58.714321 20004 ts_tablet_manager.cc:1397] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:21:58.714430 20005 log.cc:824] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:58.714602 20006 tablet_bootstrap.cc:492] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565: No bootstrap required, opened a new log
I20260927 11:21:58.714670 20006 ts_tablet_manager.cc:1397] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:21:58.715188 20005 tablet_bootstrap.cc:492] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd: No bootstrap required, opened a new log
I20260927 11:21:58.715270 20005 ts_tablet_manager.cc:1397] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:21:58.716282 20004 raft_consensus.cc:348] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253695c10c8942bfbf721550a491605a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40853 } } peers { permanent_uuid: "c1fe910373ac4c0fb642655949f72cbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41861 } } peers { permanent_uuid: "8980daae73a24c80bc90a56d59a8f565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43031 } }
I20260927 11:21:58.716463 20004 raft_consensus.cc:374] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:58.716502 20004 raft_consensus.cc:729] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 253695c10c8942bfbf721550a491605a, State: Initialized, Role: FOLLOWER
I20260927 11:21:58.716627 20004 consensus_queue.cc:260] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253695c10c8942bfbf721550a491605a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40853 } } peers { permanent_uuid: "c1fe910373ac4c0fb642655949f72cbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41861 } } peers { permanent_uuid: "8980daae73a24c80bc90a56d59a8f565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43031 } }
I20260927 11:21:58.716894 20004 ts_tablet_manager.cc:1428] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:21:58.716856 20006 raft_consensus.cc:348] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253695c10c8942bfbf721550a491605a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40853 } } peers { permanent_uuid: "c1fe910373ac4c0fb642655949f72cbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41861 } } peers { permanent_uuid: "8980daae73a24c80bc90a56d59a8f565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43031 } }
I20260927 11:21:58.716874 20005 raft_consensus.cc:348] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253695c10c8942bfbf721550a491605a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40853 } } peers { permanent_uuid: "c1fe910373ac4c0fb642655949f72cbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41861 } } peers { permanent_uuid: "8980daae73a24c80bc90a56d59a8f565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43031 } }
I20260927 11:21:58.717013 20006 raft_consensus.cc:374] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:58.717026 19987 heartbeater.cc:503] Master 127.0.98.62:35755 was elected leader, sending a full tablet report...
I20260927 11:21:58.717036 20005 raft_consensus.cc:374] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:58.717089 20006 raft_consensus.cc:729] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8980daae73a24c80bc90a56d59a8f565, State: Initialized, Role: FOLLOWER
I20260927 11:21:58.717109 20005 raft_consensus.cc:729] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1fe910373ac4c0fb642655949f72cbd, State: Initialized, Role: FOLLOWER
I20260927 11:21:58.717252 20006 consensus_queue.cc:260] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253695c10c8942bfbf721550a491605a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40853 } } peers { permanent_uuid: "c1fe910373ac4c0fb642655949f72cbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41861 } } peers { permanent_uuid: "8980daae73a24c80bc90a56d59a8f565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43031 } }
I20260927 11:21:58.717218 20005 consensus_queue.cc:260] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253695c10c8942bfbf721550a491605a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40853 } } peers { permanent_uuid: "c1fe910373ac4c0fb642655949f72cbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41861 } } peers { permanent_uuid: "8980daae73a24c80bc90a56d59a8f565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43031 } }
I20260927 11:21:58.717519 19853 heartbeater.cc:503] Master 127.0.98.62:35755 was elected leader, sending a full tablet report...
I20260927 11:21:58.717540 20006 ts_tablet_manager.cc:1428] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 11:21:58.717638 20005 ts_tablet_manager.cc:1428] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:21:58.717814 19719 heartbeater.cc:503] Master 127.0.98.62:35755 was elected leader, sending a full tablet report...
W20260927 11:21:58.768033 19854 tablet.cc:2369] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:21:58.862262 19720 tablet.cc:2369] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:21:58.866008 20012 raft_consensus.cc:482] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:21:58.866156 20012 raft_consensus.cc:504] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253695c10c8942bfbf721550a491605a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40853 } } peers { permanent_uuid: "c1fe910373ac4c0fb642655949f72cbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41861 } } peers { permanent_uuid: "8980daae73a24c80bc90a56d59a8f565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43031 } }
I20260927 11:21:58.866585 20012 leader_election.cc:288] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 253695c10c8942bfbf721550a491605a (127.0.98.3:40853), 8980daae73a24c80bc90a56d59a8f565 (127.0.98.2:43031)
I20260927 11:21:58.869642 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.866952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40679 (local address 127.0.98.2:43031)
0927 11:21:58.866998 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:58.867000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:58.867034 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:21:58.867129 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:58.867130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:58.867146 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:58.867213 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:58.867394 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.867850 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.868876 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.869272 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.869355 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.869366 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:58.869531 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 11:21:58.869573 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:21:58.869692 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.866934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53551 (local address 127.0.98.3:40853)
0927 11:21:58.866983 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:58.866986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:58.866997 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:21:58.867072 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:58.867073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:58.867082 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:58.867169 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:58.867335 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.867842 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.868871 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.869335 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.869418 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.869428 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:58.869589 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 11:21:58.869624 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 11:21:58.869689 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.866810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43031 (local address 127.0.98.1:40679)
0927 11:21:58.866968 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:58.866978 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:58.867041 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:58.867230 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:58.867231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:58.867233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:58.867319 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.867321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.867943 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.867944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.868774 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.868776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.869366 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.869367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.869446 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.869449 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:21:58.869518 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 11:21:58.869568 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":23,"thread_start_us":79,"threads_started":1}
I20260927 11:21:58.869699 19721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.866810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40853 (local address 127.0.98.1:53551)
0927 11:21:58.866899 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:58.866912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:58.866996 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:58.867176 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:58.867177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:58.867178 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:58.867260 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.867261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.867953 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.867955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.868773 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:58.868774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.869419 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.869420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:58.869502 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.869505 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:21:58.869580 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 11:21:58.869619 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":53}
I20260927 11:21:58.870077 19807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0745346f8cf4d55876038d5f4f9fff1" candidate_uuid: "c1fe910373ac4c0fb642655949f72cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8980daae73a24c80bc90a56d59a8f565" is_pre_election: true
I20260927 11:21:58.870268 19807 raft_consensus.cc:2454] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1fe910373ac4c0fb642655949f72cbd in term 0.
I20260927 11:21:58.870302 19941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0745346f8cf4d55876038d5f4f9fff1" candidate_uuid: "c1fe910373ac4c0fb642655949f72cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253695c10c8942bfbf721550a491605a" is_pre_election: true
I20260927 11:21:58.870477 19941 raft_consensus.cc:2454] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1fe910373ac4c0fb642655949f72cbd in term 0.
I20260927 11:21:58.870494 19607 leader_election.cc:302] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8980daae73a24c80bc90a56d59a8f565, c1fe910373ac4c0fb642655949f72cbd; no voters:
I20260927 11:21:58.870607 20012 raft_consensus.cc:2781] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:21:58.870671 20012 raft_consensus.cc:482] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:21:58.870702 20012 raft_consensus.cc:3037] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:58.871423 20012 raft_consensus.cc:504] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253695c10c8942bfbf721550a491605a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40853 } } peers { permanent_uuid: "c1fe910373ac4c0fb642655949f72cbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41861 } } peers { permanent_uuid: "8980daae73a24c80bc90a56d59a8f565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43031 } }
I20260927 11:21:58.871578 20012 leader_election.cc:288] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [CANDIDATE]: Term 1 election: Requested vote from peers 253695c10c8942bfbf721550a491605a (127.0.98.3:40853), 8980daae73a24c80bc90a56d59a8f565 (127.0.98.2:43031)
I20260927 11:21:58.871731 19807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0745346f8cf4d55876038d5f4f9fff1" candidate_uuid: "c1fe910373ac4c0fb642655949f72cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8980daae73a24c80bc90a56d59a8f565"
I20260927 11:21:58.871814 19807 raft_consensus.cc:3037] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:58.872505 19807 raft_consensus.cc:2454] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1fe910373ac4c0fb642655949f72cbd in term 1.
I20260927 11:21:58.872655 19941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0745346f8cf4d55876038d5f4f9fff1" candidate_uuid: "c1fe910373ac4c0fb642655949f72cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "253695c10c8942bfbf721550a491605a"
I20260927 11:21:58.872716 19607 leader_election.cc:302] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8980daae73a24c80bc90a56d59a8f565, c1fe910373ac4c0fb642655949f72cbd; no voters:
I20260927 11:21:58.872753 19941 raft_consensus.cc:3037] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:58.872815 20012 raft_consensus.cc:2781] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:21:58.873008 20012 raft_consensus.cc:686] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [term 1 LEADER]: Becoming Leader. State: Replica: c1fe910373ac4c0fb642655949f72cbd, State: Running, Role: LEADER
I20260927 11:21:58.873167 20012 consensus_queue.cc:237] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253695c10c8942bfbf721550a491605a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40853 } } peers { permanent_uuid: "c1fe910373ac4c0fb642655949f72cbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41861 } } peers { permanent_uuid: "8980daae73a24c80bc90a56d59a8f565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43031 } }
I20260927 11:21:58.873661 19941 raft_consensus.cc:2454] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1fe910373ac4c0fb642655949f72cbd in term 1.
I20260927 11:21:58.874034 19527 catalog_manager.cc:5495] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd reported cstate change: term changed from 0 to 1, leader changed from <none> to c1fe910373ac4c0fb642655949f72cbd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c1fe910373ac4c0fb642655949f72cbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "253695c10c8942bfbf721550a491605a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1fe910373ac4c0fb642655949f72cbd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8980daae73a24c80bc90a56d59a8f565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43031 } health_report { overall_health: UNKNOWN } } }
I20260927 11:21:58.891985 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.889598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.0.98.1:41861)
0927 11:21:58.889661 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:58.889663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:58.889692 (+ 29us) server_negotiation.cc:407] Connection header received
0927 11:21:58.889735 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:58.889736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:58.889762 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:58.889844 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:21:58.890008 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.890463 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.891330 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:58.891513 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:58.891581 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:58.891698 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:21:58.891800 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:21:58.891862 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:58.891893 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 11:21:58.891911 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 11:21:58.893908 19807 raft_consensus.cc:1261] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565 [term 1 FOLLOWER]: Refusing update from remote peer c1fe910373ac4c0fb642655949f72cbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:58.894299 19941 raft_consensus.cc:1261] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a [term 1 FOLLOWER]: Refusing update from remote peer c1fe910373ac4c0fb642655949f72cbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:58.894322 20014 consensus_queue.cc:1035] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8980daae73a24c80bc90a56d59a8f565" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:58.894577 20014 consensus_queue.cc:1035] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "253695c10c8942bfbf721550a491605a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:58.896818 20017 mvcc.cc:205] Tried to move back new op lower bound from 7333921254986760192 to 7333921254905249792. Current Snapshot: MvccSnapshot[applied={T|T < 7333921254986760192}]
I20260927 11:21:58.897220 20019 mvcc.cc:205] Tried to move back new op lower bound from 7333921254986760192 to 7333921254905249792. Current Snapshot: MvccSnapshot[applied={T|T < 7333921254986760192}]
I20260927 11:21:58.898311 20018 mvcc.cc:205] Tried to move back new op lower bound from 7333921254986760192 to 7333921254905249792. Current Snapshot: MvccSnapshot[applied={T|T < 7333921254986760192 or (T in {7333921254986760192})}]
I20260927 11:21:58.899438 19527 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:58.899580 19527 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:58.900300 19527 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d6241aa8d8a442288200b3743beef4a: Sending DeleteTablet for 3 replicas of tablet c0745346f8cf4d55876038d5f4f9fff1
I20260927 11:21:58.900624 19921 tablet_service.cc:1514] Processing DeleteTablet for tablet c0745346f8cf4d55876038d5f4f9fff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35368
I20260927 11:21:58.900647 19653 tablet_service.cc:1514] Processing DeleteTablet for tablet c0745346f8cf4d55876038d5f4f9fff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53062
I20260927 11:21:58.900663 19787 tablet_service.cc:1514] Processing DeleteTablet for tablet c0745346f8cf4d55876038d5f4f9fff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49126
I20260927 11:21:58.900866 20026 tablet_replica.cc:330] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a: stopping tablet replica
I20260927 11:21:58.900960 20026 raft_consensus.cc:2229] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:21:58.901031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 19590
I20260927 11:21:58.901086 20026 raft_consensus.cc:2258] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:58.901147 20027 tablet_replica.cc:330] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd: stopping tablet replica
I20260927 11:21:58.901232 20027 raft_consensus.cc:2229] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:21:58.901314 20027 raft_consensus.cc:2258] T c0745346f8cf4d55876038d5f4f9fff1 P c1fe910373ac4c0fb642655949f72cbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:58.901609 20028 tablet_replica.cc:330] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565: stopping tablet replica
I20260927 11:21:58.901603 20026 ts_tablet_manager.cc:1905] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:58.901691 20028 raft_consensus.cc:2229] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:21:58.901783 20028 raft_consensus.cc:2258] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:58.902166 20028 ts_tablet_manager.cc:1905] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:58.903541 20026 ts_tablet_manager.cc:1918] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:58.903604 20026 log.cc:1196] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/c0745346f8cf4d55876038d5f4f9fff1
I20260927 11:21:58.903892 20026 ts_tablet_manager.cc:1939] T c0745346f8cf4d55876038d5f4f9fff1 P 253695c10c8942bfbf721550a491605a: Deleting consensus metadata
I20260927 11:21:58.903965 20028 ts_tablet_manager.cc:1918] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:58.904038 20028 log.cc:1196] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507878147783-392-0/minicluster-data/ts-1/wal/wals/c0745346f8cf4d55876038d5f4f9fff1
I20260927 11:21:58.904309 20028 ts_tablet_manager.cc:1939] T c0745346f8cf4d55876038d5f4f9fff1 P 8980daae73a24c80bc90a56d59a8f565: Deleting consensus metadata
I20260927 11:21:58.904299 19514 catalog_manager.cc:4841] TS 253695c10c8942bfbf721550a491605a (127.0.98.3:40853): tablet c0745346f8cf4d55876038d5f4f9fff1 (table test-table [id=6662aa4666c942a3818dbe85584fde18]) successfully deleted
I20260927 11:21:58.904695 19513 catalog_manager.cc:4841] TS 8980daae73a24c80bc90a56d59a8f565 (127.0.98.2:43031): tablet c0745346f8cf4d55876038d5f4f9fff1 (table test-table [id=6662aa4666c942a3818dbe85584fde18]) successfully deleted
W20260927 11:21:58.907671 19513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41861 (error 108)
I20260927 11:21:58.907876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 19724
I20260927 11:21:58.908627 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:58.908528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41861 (local address 127.0.0.1:53076)
0927 11:21:58.908569 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:58.908602 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 11:21:58.908747 19513 catalog_manager.cc:4584] TS c1fe910373ac4c0fb642655949f72cbd (127.0.98.1:41861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0745346f8cf4d55876038d5f4f9fff1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41861: connect: Connection refused (error 111)
I20260927 11:21:58.912863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 19858
I20260927 11:21:58.917284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 19496
2026-09-27T11:21:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2024 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T11:21:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:21:58Z Disabled control of system clock
I20260927 11:21:58.944598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45207
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:21:59.029603 20033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:59.029847 20033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:59.029870 20033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:59.030994 20033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:21:59.031029 20033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:59.031042 20033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:59.031055 20033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:21:59.031067 20033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:21:59.032368 20033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45207
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:59.032575 20033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:59.032819 20033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:59.037717 20039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:59.037868 20033 server_base.cc:1034] running on GCE node
W20260927 11:21:59.037726 20041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:59.037745 20038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:59.038226 20033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:59.038501 20033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:59.039696 20033 hybrid_clock.cc:648] HybridClock initialized: now 1790508119039688 us; error 37 us; skew 500 ppm
I20260927 11:21:59.041038 20033 webserver.cc:460] Webserver started at http://127.0.98.62:38863/ using document root <none> and password file <none>
I20260927 11:21:59.041353 20033 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:59.041424 20033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:59.041587 20033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:59.042696 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "3925822bac434b43b1a09587794dd22b"
format_stamp: "Formatted at 2026-09-27 11:21:59 on dist-test-slave-lsvg"
server_key: "5f7638803f22d6eda52ad0e37280ad8d"
server_key_iv: "2212479e4597529b4832a9ca57b9a6d4"
server_key_version: "encryptionkey@0"
I20260927 11:21:59.043087 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "3925822bac434b43b1a09587794dd22b"
format_stamp: "Formatted at 2026-09-27 11:21:59 on dist-test-slave-lsvg"
server_key: "5f7638803f22d6eda52ad0e37280ad8d"
server_key_iv: "2212479e4597529b4832a9ca57b9a6d4"
server_key_version: "encryptionkey@0"
I20260927 11:21:59.044435 20033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:21:59.045190 20047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:59.045363 20033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:21:59.045470 20033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "3925822bac434b43b1a09587794dd22b"
format_stamp: "Formatted at 2026-09-27 11:21:59 on dist-test-slave-lsvg"
server_key: "5f7638803f22d6eda52ad0e37280ad8d"
server_key_iv: "2212479e4597529b4832a9ca57b9a6d4"
server_key_version: "encryptionkey@0"
I20260927 11:21:59.045558 20033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:59.055639 20033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:59.055986 20033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:59.056146 20033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:59.060441 20033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45207
I20260927 11:21:59.060470 20099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45207 every 8 connection(s)
I20260927 11:21:59.060832 20033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:21:59.061877 20100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:59.064309 20100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b: Bootstrap starting.
I20260927 11:21:59.064935 20100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:59.065282 20100 log.cc:824] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:59.066001 20100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b: No bootstrap required, opened a new log
I20260927 11:21:59.067607 20100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3925822bac434b43b1a09587794dd22b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45207 } }
I20260927 11:21:59.067750 20100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:59.067772 20100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3925822bac434b43b1a09587794dd22b, State: Initialized, Role: FOLLOWER
I20260927 11:21:59.067962 20100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3925822bac434b43b1a09587794dd22b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45207 } }
I20260927 11:21:59.068032 20100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:21:59.068053 20100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:21:59.068127 20100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:59.068747 20100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3925822bac434b43b1a09587794dd22b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45207 } }
I20260927 11:21:59.068774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 20033
I20260927 11:21:59.068876 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:21:59.068869 20100 leader_election.cc:302] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3925822bac434b43b1a09587794dd22b; no voters:
I20260927 11:21:59.069036 20100 leader_election.cc:288] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:21:59.069154 392 external_mini_cluster.cc:1428] Setting key 755c12aa1508fcc78f00fac958aa87a7
I20260927 11:21:59.069235 20103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:21:59.069356 20100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:21:59.069424 20103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [term 1 LEADER]: Becoming Leader. State: Replica: 3925822bac434b43b1a09587794dd22b, State: Running, Role: LEADER
I20260927 11:21:59.069613 20103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3925822bac434b43b1a09587794dd22b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45207 } }
I20260927 11:21:59.070156 20104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3925822bac434b43b1a09587794dd22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3925822bac434b43b1a09587794dd22b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45207 } } }
I20260927 11:21:59.070240 20104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [sys.catalog]: This master's current role is: LEADER
I20260927 11:21:59.070452 20105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3925822bac434b43b1a09587794dd22b. Latest consensus state: current_term: 1 leader_uuid: "3925822bac434b43b1a09587794dd22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3925822bac434b43b1a09587794dd22b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45207 } } }
I20260927 11:21:59.070487 20114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:21:59.070513 20105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b [sys.catalog]: This master's current role is: LEADER
I20260927 11:21:59.070983 20114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:21:59.072551 20114 catalog_manager.cc:1347] Generated new cluster ID: 39405500b7bd4629b9db248b06ee1f7b
I20260927 11:21:59.072610 20114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:21:59.075680 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.069813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.0.98.62:45207)
0927 11:21:59.070017 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:59.070028 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:59.070063 (+ 35us) server_negotiation.cc:407] Connection header received
0927 11:21:59.070199 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:59.070208 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:59.070290 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:59.070726 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:59.070838 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.072275 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.073448 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.074071 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.074204 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.074948 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:59.074952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:59.074952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:59.075166 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:59.075210 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:59.075225 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:59.075295 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:59.075365 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 11:21:59.075448 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260927 11:21:59.099866 20114 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:21:59.100492 20114 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:21:59.107829 20114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b: Generated new TSK 0
I20260927 11:21:59.108078 20114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:21:59.113034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45207
--builtin_ntp_servers=127.0.98.20:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:21:59.203231 20124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:59.203464 20124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:59.203490 20124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:59.204952 20124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:59.204999 20124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:59.205027 20124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:21:59.206439 20124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:59.206705 20124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:59.207028 20124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:59.212037 20130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:59.212109 20132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:59.212105 20129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:59.212430 20124 server_base.cc:1034] running on GCE node
I20260927 11:21:59.212704 20124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:59.212975 20124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:59.214143 20124 hybrid_clock.cc:648] HybridClock initialized: now 1790508119214116 us; error 60 us; skew 500 ppm
I20260927 11:21:59.215530 20124 webserver.cc:460] Webserver started at http://127.0.98.1:46311/ using document root <none> and password file <none>
I20260927 11:21:59.215783 20124 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:59.215830 20124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:59.215974 20124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:59.217134 20124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/data/instance:
uuid: "38929e765ca54ae39b65cfa547413ca5"
format_stamp: "Formatted at 2026-09-27 11:21:59 on dist-test-slave-lsvg"
server_key: "29cea343e1d7ac621c9f868f4651874b"
server_key_iv: "6ed5e59726a2de6d525fb5ab4499ab26"
server_key_version: "encryptionkey@0"
I20260927 11:21:59.217497 20124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/wal/instance:
uuid: "38929e765ca54ae39b65cfa547413ca5"
format_stamp: "Formatted at 2026-09-27 11:21:59 on dist-test-slave-lsvg"
server_key: "29cea343e1d7ac621c9f868f4651874b"
server_key_iv: "6ed5e59726a2de6d525fb5ab4499ab26"
server_key_version: "encryptionkey@0"
I20260927 11:21:59.218860 20124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:21:59.219686 20138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:59.219959 20124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:21:59.220041 20124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/wal
uuid: "38929e765ca54ae39b65cfa547413ca5"
format_stamp: "Formatted at 2026-09-27 11:21:59 on dist-test-slave-lsvg"
server_key: "29cea343e1d7ac621c9f868f4651874b"
server_key_iv: "6ed5e59726a2de6d525fb5ab4499ab26"
server_key_version: "encryptionkey@0"
I20260927 11:21:59.220150 20124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:59.240507 20124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:59.240859 20124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:59.241040 20124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:59.241338 20124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:59.241693 20124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:59.241747 20124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:59.241783 20124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:59.241811 20124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:59.247691 20124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46597
I20260927 11:21:59.247732 20251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46597 every 8 connection(s)
I20260927 11:21:59.248127 20124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:21:59.248766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 20124
I20260927 11:21:59.248889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:21:59.249203 392 external_mini_cluster.cc:1428] Setting key 03e48969cbfd864836b5aca56c7bad61
I20260927 11:21:59.250308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45207
--builtin_ntp_servers=127.0.98.20:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 11:21:59.254356 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.248907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43177 (local address 127.0.98.62:45207)
0927 11:21:59.248958 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:59.248960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:59.249342 (+ 382us) server_negotiation.cc:407] Connection header received
0927 11:21:59.250097 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:59.250098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:59.250139 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:59.250231 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:59.251049 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.251826 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.252824 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.253096 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.253170 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.254076 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:59.254078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:59.254078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:59.254083 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:59.254097 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:59.254102 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:59.254153 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:59.254243 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 11:21:59.254277 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 11:21:59.254587 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.248922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45207 (local address 127.0.98.1:43177)
0927 11:21:59.249181 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:59.249249 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:59.250001 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:59.250315 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:59.250319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:59.250336 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:59.250941 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.250947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.251934 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.251935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.252740 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.252743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.253192 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.253193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.253336 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.253914 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:21:59.253914 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:59.253944 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:21:59.253966 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:59.254165 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:59.254170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:59.254172 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:21:59.254243 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:21:59.254345 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":143,"threads_started":1}
I20260927 11:21:59.255003 20252 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45207
I20260927 11:21:59.255116 20252 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:59.255300 20252 heartbeater.cc:511] Master 127.0.98.62:45207 requested a full tablet report, sending...
I20260927 11:21:59.255795 20064 ts_manager.cc:194] Registered new tserver with Master: 38929e765ca54ae39b65cfa547413ca5 (127.0.98.1:46597)
I20260927 11:21:59.256364 20064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:43177
W20260927 11:21:59.333184 20255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:59.333426 20255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:59.333460 20255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:59.334657 20255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:59.334722 20255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:59.334774 20255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:21:59.336165 20255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:59.336440 20255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:59.336755 20255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:59.342074 20260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:59.342150 20263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:59.342212 20255 server_base.cc:1034] running on GCE node
W20260927 11:21:59.342074 20261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:59.342469 20255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:59.342697 20255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:59.343842 20255 hybrid_clock.cc:648] HybridClock initialized: now 1790508119343828 us; error 27 us; skew 500 ppm
I20260927 11:21:59.345270 20255 webserver.cc:460] Webserver started at http://127.0.98.2:43915/ using document root <none> and password file <none>
I20260927 11:21:59.345505 20255 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:59.345561 20255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:59.345726 20255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:59.346804 20255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/data/instance:
uuid: "636d4036fc8548058aeb7704b6963942"
format_stamp: "Formatted at 2026-09-27 11:21:59 on dist-test-slave-lsvg"
server_key: "f9948e05f49f0c892d0452bf9fd85940"
server_key_iv: "a735424bd5db7aad999446fe1f87bdf8"
server_key_version: "encryptionkey@0"
I20260927 11:21:59.347160 20255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/wal/instance:
uuid: "636d4036fc8548058aeb7704b6963942"
format_stamp: "Formatted at 2026-09-27 11:21:59 on dist-test-slave-lsvg"
server_key: "f9948e05f49f0c892d0452bf9fd85940"
server_key_iv: "a735424bd5db7aad999446fe1f87bdf8"
server_key_version: "encryptionkey@0"
I20260927 11:21:59.348546 20255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:21:59.349503 20269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:59.349720 20255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:21:59.349826 20255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/wal
uuid: "636d4036fc8548058aeb7704b6963942"
format_stamp: "Formatted at 2026-09-27 11:21:59 on dist-test-slave-lsvg"
server_key: "f9948e05f49f0c892d0452bf9fd85940"
server_key_iv: "a735424bd5db7aad999446fe1f87bdf8"
server_key_version: "encryptionkey@0"
I20260927 11:21:59.349908 20255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:59.369164 20255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:59.369613 20255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:59.369819 20255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:59.370077 20255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:59.370427 20255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:59.370497 20255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:59.370530 20255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:59.370553 20255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:59.376207 20255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46883
I20260927 11:21:59.376266 20382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46883 every 8 connection(s)
I20260927 11:21:59.376618 20255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:21:59.382151 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.377488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37561 (local address 127.0.98.62:45207)
0927 11:21:59.377581 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:59.377583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:59.377807 (+ 224us) server_negotiation.cc:407] Connection header received
0927 11:21:59.378330 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:59.378331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:59.378342 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:59.378402 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:59.379164 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.379671 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.380698 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.380881 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.380933 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.381896 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:59.381897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:59.381897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:59.381901 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:59.381915 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:59.381919 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:59.381973 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:59.382054 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 11:21:59.382081 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 11:21:59.382333 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.377452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45207 (local address 127.0.98.2:37561)
0927 11:21:59.377668 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:59.377705 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:59.378220 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:59.378532 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:59.378536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:59.378545 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:59.379059 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.379063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.379761 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.379762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.380615 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.380617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.380960 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.380961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.381133 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.381755 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:21:59.381755 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:59.381787 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:21:59.381811 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:59.381982 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:59.381986 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:59.381988 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:21:59.382040 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 11:21:59.382124 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":108,"threads_started":1}
I20260927 11:21:59.382854 20383 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45207
I20260927 11:21:59.382983 20383 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:59.383189 20383 heartbeater.cc:511] Master 127.0.98.62:45207 requested a full tablet report, sending...
I20260927 11:21:59.383630 20064 ts_manager.cc:194] Registered new tserver with Master: 636d4036fc8548058aeb7704b6963942 (127.0.98.2:46883)
I20260927 11:21:59.384132 20064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37561
I20260927 11:21:59.386032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 20255
I20260927 11:21:59.386129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:21:59.386294 392 external_mini_cluster.cc:1428] Setting key d3bea42fdeb526a3072e7895b5f2736a
I20260927 11:21:59.387342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45207
--builtin_ntp_servers=127.0.98.20:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:21:59.472886 20386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:21:59.473167 20386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:21:59.473194 20386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:21:59.474311 20386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:21:59.474345 20386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:21:59.474370 20386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:21:59.475638 20386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:21:59.475838 20386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:21:59.476092 20386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:21:59.481117 20394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:21:59.481196 20391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:59.481310 20386 server_base.cc:1034] running on GCE node
W20260927 11:21:59.481129 20392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:21:59.481657 20386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:21:59.481917 20386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:21:59.483090 20386 hybrid_clock.cc:648] HybridClock initialized: now 1790508119483077 us; error 31 us; skew 500 ppm
I20260927 11:21:59.484462 20386 webserver.cc:460] Webserver started at http://127.0.98.3:35251/ using document root <none> and password file <none>
I20260927 11:21:59.484719 20386 fs_manager.cc:359] Metadata directory not provided
I20260927 11:21:59.484791 20386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:21:59.484938 20386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:21:59.486114 20386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/data/instance:
uuid: "560d7c0020da40f8a5348d3d4b4096b1"
format_stamp: "Formatted at 2026-09-27 11:21:59 on dist-test-slave-lsvg"
server_key: "71ab5611201c96817e6481e562524cea"
server_key_iv: "03dc56cd308bf5b1f9614a16e793c503"
server_key_version: "encryptionkey@0"
I20260927 11:21:59.486479 20386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/wal/instance:
uuid: "560d7c0020da40f8a5348d3d4b4096b1"
format_stamp: "Formatted at 2026-09-27 11:21:59 on dist-test-slave-lsvg"
server_key: "71ab5611201c96817e6481e562524cea"
server_key_iv: "03dc56cd308bf5b1f9614a16e793c503"
server_key_version: "encryptionkey@0"
I20260927 11:21:59.487818 20386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:21:59.488714 20400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:59.488947 20386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:21:59.489081 20386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/wal
uuid: "560d7c0020da40f8a5348d3d4b4096b1"
format_stamp: "Formatted at 2026-09-27 11:21:59 on dist-test-slave-lsvg"
server_key: "71ab5611201c96817e6481e562524cea"
server_key_iv: "03dc56cd308bf5b1f9614a16e793c503"
server_key_version: "encryptionkey@0"
I20260927 11:21:59.489168 20386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:21:59.521893 20386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:21:59.522257 20386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:21:59.522461 20386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:21:59.522728 20386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:21:59.523085 20386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:21:59.523140 20386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:59.523185 20386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:21:59.523211 20386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:21:59.528944 20386 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33483
I20260927 11:21:59.529024 20513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33483 every 8 connection(s)
I20260927 11:21:59.529392 20386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:21:59.532521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 20386
I20260927 11:21:59.532627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:21:59.532836 392 external_mini_cluster.cc:1428] Setting key 5b817c3b0a36bcab544eabcf487866c0
I20260927 11:21:59.535192 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.530136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45523 (local address 127.0.98.62:45207)
0927 11:21:59.530213 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:59.530216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:59.530422 (+ 206us) server_negotiation.cc:407] Connection header received
0927 11:21:59.530947 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:59.530949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:59.530973 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:59.531048 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:59.531857 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.532412 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.533831 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.534053 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.534110 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.534970 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:59.534971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:59.534972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:59.534976 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:59.534990 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:59.534993 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:59.535039 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:59.535109 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 11:21:59.535135 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 11:21:59.535460 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.530116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45207 (local address 127.0.98.3:45523)
0927 11:21:59.530293 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:59.530335 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:59.530843 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:59.531226 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:59.531229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:59.531238 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:21:59.531770 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.531775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.532500 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.532501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.533742 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.533745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.534123 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.534124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.534232 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.534831 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:21:59.534831 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:21:59.534860 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:21:59.534882 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:21:59.535051 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:21:59.535056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:21:59.535058 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:21:59.535113 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 11:21:59.535192 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260927 11:21:59.535976 20514 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45207
I20260927 11:21:59.536079 20514 heartbeater.cc:464] Registering TS with master...
I20260927 11:21:59.536274 20514 heartbeater.cc:511] Master 127.0.98.62:45207 requested a full tablet report, sending...
I20260927 11:21:59.536674 20064 ts_manager.cc:194] Registered new tserver with Master: 560d7c0020da40f8a5348d3d4b4096b1 (127.0.98.3:33483)
I20260927 11:21:59.537099 20064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45523
I20260927 11:21:59.544751 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:21:59.549609 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.546356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.0.98.62:45207)
0927 11:21:59.546447 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:59.546450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:59.546526 (+ 76us) server_negotiation.cc:407] Connection header received
0927 11:21:59.546599 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:59.546600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:59.546614 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:59.546685 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:21:59.546866 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.547366 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.548267 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.548491 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.548566 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.549333 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:21:59.549334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:21:59.549334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:21:59.549340 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:21:59.549354 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:21:59.549361 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:21:59.549424 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:59.549488 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 11:21:59.549516 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 11:21:59.551296 20064 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:21:59.553103 20064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:21:59.559921 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.555299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.0.98.3:33483)
0927 11:21:59.555449 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:59.555454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:59.555745 (+ 291us) server_negotiation.cc:407] Connection header received
0927 11:21:59.555844 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:59.555846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:59.555863 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:59.555952 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:59.556323 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.557192 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.558647 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.559448 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.559531 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.559736 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:59.559773 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 11:21:59.559826 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260927 11:21:59.559923 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.555454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50180 (local address 127.0.98.1:46597)
0927 11:21:59.555674 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:59.555678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:59.556083 (+ 405us) server_negotiation.cc:407] Connection header received
0927 11:21:59.556130 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:59.556132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:59.556147 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:59.556223 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:59.556352 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.557510 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.558780 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.559466 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.559530 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.559735 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:59.559773 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 11:21:59.559817 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
I20260927 11:21:59.559901 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.555228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46597 (local address 127.0.0.1:50180)
0927 11:21:59.555878 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:59.555887 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:59.555966 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:59.556234 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:59.556235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:59.556237 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:59.556281 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.556282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.557598 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.557599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.558699 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.558702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.559550 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.559551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.559635 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.559650 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 11:21:59.559717 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 11:21:59.559798 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":51,"thread_start_us":74,"threads_started":1}
I20260927 11:21:59.559901 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.555220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33483 (local address 127.0.0.1:46410)
0927 11:21:59.555638 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:59.555682 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:59.555763 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:59.556034 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:59.556037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:59.556039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:59.556134 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.556135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.557282 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.557283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.558553 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.558555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.559549 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.559550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.559635 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.559650 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 11:21:59.559717 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 11:21:59.559798 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":4,"thread_start_us":60,"threads_started":1}
I20260927 11:21:59.560806 20186 tablet_service.cc:1467] Processing CreateTablet for tablet e1da8245868341848633bb9710e9a27f (DEFAULT_TABLE table=test-table [id=8a9aa196a7964d9ab5d9488417575fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:59.561211 20186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1da8245868341848633bb9710e9a27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:59.561626 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.555222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46883 (local address 127.0.0.1:33256)
0927 11:21:59.555657 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:59.555682 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:59.555763 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:59.556301 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:59.556302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:59.556305 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:59.556358 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.556360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.557642 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.557643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.558565 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.558567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.561346 (+ 2779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.561347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.561433 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.561437 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:21:59.561492 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 11:21:59.561542 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":2,"spinlock_wait_cycles":896,"thread_start_us":88,"threads_started":1}
I20260927 11:21:59.561643 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.555450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33256 (local address 127.0.98.2:46883)
0927 11:21:59.555648 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:59.555653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:59.555739 (+ 86us) server_negotiation.cc:407] Connection header received
0927 11:21:59.555844 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:59.555846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:59.555863 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:59.555951 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:59.556426 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.557581 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.560580 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.561227 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.561340 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.561491 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:59.561521 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 11:21:59.561561 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260927 11:21:59.564147 20528 tablet_bootstrap.cc:492] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5: Bootstrap starting.
I20260927 11:21:59.564457 20448 tablet_service.cc:1467] Processing CreateTablet for tablet e1da8245868341848633bb9710e9a27f (DEFAULT_TABLE table=test-table [id=8a9aa196a7964d9ab5d9488417575fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:59.564769 20448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1da8245868341848633bb9710e9a27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:59.564932 20528 tablet_bootstrap.cc:654] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:59.565343 20528 log.cc:824] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:59.565816 20317 tablet_service.cc:1467] Processing CreateTablet for tablet e1da8245868341848633bb9710e9a27f (DEFAULT_TABLE table=test-table [id=8a9aa196a7964d9ab5d9488417575fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:21:59.566171 20317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1da8245868341848633bb9710e9a27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:21:59.566236 20528 tablet_bootstrap.cc:492] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5: No bootstrap required, opened a new log
I20260927 11:21:59.566313 20528 ts_tablet_manager.cc:1397] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 11:21:59.568087 20530 tablet_bootstrap.cc:492] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1: Bootstrap starting.
I20260927 11:21:59.568518 20528 raft_consensus.cc:348] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4036fc8548058aeb7704b6963942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46883 } } peers { permanent_uuid: "560d7c0020da40f8a5348d3d4b4096b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33483 } } peers { permanent_uuid: "38929e765ca54ae39b65cfa547413ca5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46597 } }
I20260927 11:21:59.568712 20528 raft_consensus.cc:374] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:59.568764 20528 raft_consensus.cc:729] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38929e765ca54ae39b65cfa547413ca5, State: Initialized, Role: FOLLOWER
I20260927 11:21:59.568914 20528 consensus_queue.cc:260] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4036fc8548058aeb7704b6963942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46883 } } peers { permanent_uuid: "560d7c0020da40f8a5348d3d4b4096b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33483 } } peers { permanent_uuid: "38929e765ca54ae39b65cfa547413ca5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46597 } }
I20260927 11:21:59.569276 20528 ts_tablet_manager.cc:1428] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 11:21:59.569002 20530 tablet_bootstrap.cc:654] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:59.569432 20252 heartbeater.cc:503] Master 127.0.98.62:45207 was elected leader, sending a full tablet report...
I20260927 11:21:59.569614 20532 tablet_bootstrap.cc:492] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942: Bootstrap starting.
I20260927 11:21:59.570147 20530 log.cc:824] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:59.570288 20532 tablet_bootstrap.cc:654] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942: Neither blocks nor log segments found. Creating new log.
I20260927 11:21:59.570644 20532 log.cc:824] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942: Log is configured to *not* fsync() on all Append() calls
I20260927 11:21:59.571004 20530 tablet_bootstrap.cc:492] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1: No bootstrap required, opened a new log
I20260927 11:21:59.571087 20530 ts_tablet_manager.cc:1397] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 11:21:59.571520 20532 tablet_bootstrap.cc:492] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942: No bootstrap required, opened a new log
I20260927 11:21:59.571604 20532 ts_tablet_manager.cc:1397] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:21:59.573339 20530 raft_consensus.cc:348] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4036fc8548058aeb7704b6963942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46883 } } peers { permanent_uuid: "560d7c0020da40f8a5348d3d4b4096b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33483 } } peers { permanent_uuid: "38929e765ca54ae39b65cfa547413ca5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46597 } }
I20260927 11:21:59.573561 20530 raft_consensus.cc:374] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:59.573439 20532 raft_consensus.cc:348] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4036fc8548058aeb7704b6963942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46883 } } peers { permanent_uuid: "560d7c0020da40f8a5348d3d4b4096b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33483 } } peers { permanent_uuid: "38929e765ca54ae39b65cfa547413ca5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46597 } }
I20260927 11:21:59.573614 20532 raft_consensus.cc:374] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:21:59.573613 20530 raft_consensus.cc:729] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560d7c0020da40f8a5348d3d4b4096b1, State: Initialized, Role: FOLLOWER
I20260927 11:21:59.573652 20532 raft_consensus.cc:729] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 636d4036fc8548058aeb7704b6963942, State: Initialized, Role: FOLLOWER
I20260927 11:21:59.573764 20530 consensus_queue.cc:260] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4036fc8548058aeb7704b6963942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46883 } } peers { permanent_uuid: "560d7c0020da40f8a5348d3d4b4096b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33483 } } peers { permanent_uuid: "38929e765ca54ae39b65cfa547413ca5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46597 } }
I20260927 11:21:59.573779 20532 consensus_queue.cc:260] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4036fc8548058aeb7704b6963942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46883 } } peers { permanent_uuid: "560d7c0020da40f8a5348d3d4b4096b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33483 } } peers { permanent_uuid: "38929e765ca54ae39b65cfa547413ca5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46597 } }
I20260927 11:21:59.574049 20532 ts_tablet_manager.cc:1428] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:21:59.574067 20383 heartbeater.cc:503] Master 127.0.98.62:45207 was elected leader, sending a full tablet report...
I20260927 11:21:59.574049 20530 ts_tablet_manager.cc:1428] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:21:59.574067 20514 heartbeater.cc:503] Master 127.0.98.62:45207 was elected leader, sending a full tablet report...
I20260927 11:21:59.585094 20536 raft_consensus.cc:482] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:21:59.585239 20536 raft_consensus.cc:504] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4036fc8548058aeb7704b6963942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46883 } } peers { permanent_uuid: "560d7c0020da40f8a5348d3d4b4096b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33483 } } peers { permanent_uuid: "38929e765ca54ae39b65cfa547413ca5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46597 } }
I20260927 11:21:59.585633 20536 leader_election.cc:288] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 636d4036fc8548058aeb7704b6963942 (127.0.98.2:46883), 38929e765ca54ae39b65cfa547413ca5 (127.0.98.1:46597)
I20260927 11:21:59.588348 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.585752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46883 (local address 127.0.98.3:41781)
0927 11:21:59.585814 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:59.585826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:59.585882 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:59.586018 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:59.586019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:59.586020 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:59.586099 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.586101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.586796 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.586797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.587551 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.587553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.588101 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.588102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.588170 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.588175 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:21:59.588222 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 11:21:59.588252 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 11:21:59.588361 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.585843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41781 (local address 127.0.98.2:46883)
0927 11:21:59.585889 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:59.585891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:59.585903 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:21:59.585936 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:59.585937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:59.585950 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:59.586008 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:59.586165 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.586720 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.587650 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.588022 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.588106 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.588117 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:59.588243 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 11:21:59.588292 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:21:59.588421 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.585859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54547 (local address 127.0.98.1:46597)
0927 11:21:59.585903 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:59.585905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:59.586011 (+ 106us) server_negotiation.cc:407] Connection header received
0927 11:21:59.586078 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:59.586079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:59.586090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:59.586152 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:21:59.586292 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.586745 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.587683 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.588026 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.588121 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.588130 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:59.588305 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 11:21:59.588349 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:21:59.588442 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.585764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46597 (local address 127.0.98.3:54547)
0927 11:21:59.585919 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:59.585929 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:21:59.586005 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:21:59.586163 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:21:59.586164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:21:59.586165 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:21:59.586227 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.586228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.586850 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.586851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.587567 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:21:59.587569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.588126 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.588127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:21:59.588227 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.588231 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:21:59.588286 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 11:21:59.588318 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260927 11:21:59.588783 20206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1da8245868341848633bb9710e9a27f" candidate_uuid: "560d7c0020da40f8a5348d3d4b4096b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38929e765ca54ae39b65cfa547413ca5" is_pre_election: true
I20260927 11:21:59.588966 20206 raft_consensus.cc:2454] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 560d7c0020da40f8a5348d3d4b4096b1 in term 0.
I20260927 11:21:59.588946 20337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1da8245868341848633bb9710e9a27f" candidate_uuid: "560d7c0020da40f8a5348d3d4b4096b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "636d4036fc8548058aeb7704b6963942" is_pre_election: true
I20260927 11:21:59.589131 20337 raft_consensus.cc:2454] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 560d7c0020da40f8a5348d3d4b4096b1 in term 0.
I20260927 11:21:59.589226 20401 leader_election.cc:302] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38929e765ca54ae39b65cfa547413ca5, 560d7c0020da40f8a5348d3d4b4096b1; no voters:
I20260927 11:21:59.589375 20536 raft_consensus.cc:2781] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:21:59.589433 20536 raft_consensus.cc:482] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:21:59.589450 20536 raft_consensus.cc:3037] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:59.590160 20536 raft_consensus.cc:504] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4036fc8548058aeb7704b6963942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46883 } } peers { permanent_uuid: "560d7c0020da40f8a5348d3d4b4096b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33483 } } peers { permanent_uuid: "38929e765ca54ae39b65cfa547413ca5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46597 } }
I20260927 11:21:59.590297 20536 leader_election.cc:288] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [CANDIDATE]: Term 1 election: Requested vote from peers 636d4036fc8548058aeb7704b6963942 (127.0.98.2:46883), 38929e765ca54ae39b65cfa547413ca5 (127.0.98.1:46597)
I20260927 11:21:59.590438 20337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1da8245868341848633bb9710e9a27f" candidate_uuid: "560d7c0020da40f8a5348d3d4b4096b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "636d4036fc8548058aeb7704b6963942"
I20260927 11:21:59.590437 20206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1da8245868341848633bb9710e9a27f" candidate_uuid: "560d7c0020da40f8a5348d3d4b4096b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38929e765ca54ae39b65cfa547413ca5"
I20260927 11:21:59.590509 20206 raft_consensus.cc:3037] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:59.590508 20337 raft_consensus.cc:3037] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:21:59.591409 20206 raft_consensus.cc:2454] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 560d7c0020da40f8a5348d3d4b4096b1 in term 1.
I20260927 11:21:59.591421 20337 raft_consensus.cc:2454] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 560d7c0020da40f8a5348d3d4b4096b1 in term 1.
I20260927 11:21:59.591588 20404 leader_election.cc:302] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 560d7c0020da40f8a5348d3d4b4096b1, 636d4036fc8548058aeb7704b6963942; no voters:
I20260927 11:21:59.591735 20536 raft_consensus.cc:2781] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:21:59.591866 20536 raft_consensus.cc:686] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [term 1 LEADER]: Becoming Leader. State: Replica: 560d7c0020da40f8a5348d3d4b4096b1, State: Running, Role: LEADER
I20260927 11:21:59.591993 20536 consensus_queue.cc:237] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4036fc8548058aeb7704b6963942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46883 } } peers { permanent_uuid: "560d7c0020da40f8a5348d3d4b4096b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33483 } } peers { permanent_uuid: "38929e765ca54ae39b65cfa547413ca5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46597 } }
I20260927 11:21:59.592821 20064 catalog_manager.cc:5495] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 560d7c0020da40f8a5348d3d4b4096b1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "560d7c0020da40f8a5348d3d4b4096b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "636d4036fc8548058aeb7704b6963942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "560d7c0020da40f8a5348d3d4b4096b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38929e765ca54ae39b65cfa547413ca5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46597 } health_report { overall_health: UNKNOWN } } }
I20260927 11:21:59.607571 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.605111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46420 (local address 127.0.98.3:33483)
0927 11:21:59.605166 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:21:59.605168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:21:59.605215 (+ 47us) server_negotiation.cc:407] Connection header received
0927 11:21:59.605276 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:21:59.605277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:21:59.605298 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:21:59.605367 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:21:59.605503 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.605979 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.606763 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:21:59.606954 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:21:59.607029 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:21:59.607207 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:21:59.607312 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:21:59.607409 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 11:21:59.607433 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 11:21:59.607450 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:21:59.609418 20206 raft_consensus.cc:1261] T e1da8245868341848633bb9710e9a27f P 38929e765ca54ae39b65cfa547413ca5 [term 1 FOLLOWER]: Refusing update from remote peer 560d7c0020da40f8a5348d3d4b4096b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:59.609418 20337 raft_consensus.cc:1261] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942 [term 1 FOLLOWER]: Refusing update from remote peer 560d7c0020da40f8a5348d3d4b4096b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:21:59.609799 20536 consensus_queue.cc:1035] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "636d4036fc8548058aeb7704b6963942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:59.609973 20538 consensus_queue.cc:1035] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38929e765ca54ae39b65cfa547413ca5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:21:59.612532 20542 mvcc.cc:205] Tried to move back new op lower bound from 7333921257917427712 to 7333921257849516032. Current Snapshot: MvccSnapshot[applied={T|T < 7333921257917427712}]
I20260927 11:21:59.614527 20064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:59.614624 20064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:21:59.615304 20064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3925822bac434b43b1a09587794dd22b: Sending DeleteTablet for 3 replicas of tablet e1da8245868341848633bb9710e9a27f
I20260927 11:21:59.615631 20317 tablet_service.cc:1514] Processing DeleteTablet for tablet e1da8245868341848633bb9710e9a27f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:59 UTC) from {username='slave'} at 127.0.0.1:33256
I20260927 11:21:59.615659 20186 tablet_service.cc:1514] Processing DeleteTablet for tablet e1da8245868341848633bb9710e9a27f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:59 UTC) from {username='slave'} at 127.0.0.1:50180
I20260927 11:21:59.615965 20448 tablet_service.cc:1514] Processing DeleteTablet for tablet e1da8245868341848633bb9710e9a27f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:21:59 UTC) from {username='slave'} at 127.0.0.1:46410
I20260927 11:21:59.616148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 20124
I20260927 11:21:59.621212 20551 tablet_replica.cc:330] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942: stopping tablet replica
I20260927 11:21:59.621366 20551 raft_consensus.cc:2229] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:21:59.621520 20551 raft_consensus.cc:2258] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:59.621701 20552 tablet_replica.cc:330] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1: stopping tablet replica
I20260927 11:21:59.621804 20552 raft_consensus.cc:2229] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:21:59.621904 20552 raft_consensus.cc:2258] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:21:59.622366 20551 ts_tablet_manager.cc:1905] T e1da8245868341848633bb9710e9a27f P 636d4036fc8548058aeb7704b6963942: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 11:21:59.622588 20048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46597 (error 108)
I20260927 11:21:59.623077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 20255
I20260927 11:21:59.623124 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.622964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46597 (local address 127.0.0.1:50196)
0927 11:21:59.623038 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:59.623085 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 11:21:59.624027 20048 catalog_manager.cc:4584] TS 38929e765ca54ae39b65cfa547413ca5 (127.0.98.1:46597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1da8245868341848633bb9710e9a27f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46597: connect: Connection refused (error 111)
I20260927 11:21:59.624467 20552 ts_tablet_manager.cc:1905] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:21:59.626605 20552 ts_tablet_manager.cc:1918] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:21:59.626744 20552 log.cc:1196] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1: Deleting WAL directory at /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507878147783-392-0/minicluster-data/ts-2/wal/wals/e1da8245868341848633bb9710e9a27f
I20260927 11:21:59.627064 20552 ts_tablet_manager.cc:1939] T e1da8245868341848633bb9710e9a27f P 560d7c0020da40f8a5348d3d4b4096b1: Deleting consensus metadata
I20260927 11:21:59.627610 20049 catalog_manager.cc:4841] TS 560d7c0020da40f8a5348d3d4b4096b1 (127.0.98.3:33483): tablet e1da8245868341848633bb9710e9a27f (table test-table [id=8a9aa196a7964d9ab5d9488417575fe1]) successfully deleted
I20260927 11:21:59.628327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 20386
I20260927 11:21:59.628347 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:21:59.628235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46883 (local address 127.0.0.1:33258)
0927 11:21:59.628287 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 11:21:59.628310 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 11:21:59.633430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 20033
2026-09-27T11:21:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (715 ms)
[----------] 6 tests from EncryptionPolicyTest (8304 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:21:59 dist-test-slave-lsvg krb5kdc[20556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:21:59 dist-test-slave-lsvg krb5kdc[20556](info): set up 2 sockets
Sep 27 11:21:59 dist-test-slave-lsvg krb5kdc[20556](info): commencing operation
krb5kdc: starting...
W20260927 11:22:00.673034 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:00 dist-test-slave-lsvg krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:22:00.793212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:00.874287 20569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:00.874496 20569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:00.874519 20569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:00.875608 20569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:00.875644 20569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:00.875658 20569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:00.875671 20569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:00.875684 20569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:00.877195 20569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:00.877411 20569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:00.877588 20569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:00.882488 20577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:00.882510 20574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:00.882771 20569 server_base.cc:1034] running on GCE node
W20260927 11:22:00.882591 20575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:00.883049 20569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:00.883308 20569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:00.884472 20569 hybrid_clock.cc:648] HybridClock initialized: now 1790508120884470 us; error 38 us; skew 500 ppm
Sep 27 11:22:00 dist-test-slave-lsvg krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:00.886981 20569 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:22:00.887544 20569 webserver.cc:460] Webserver started at http://127.0.0.1:39809/ using document root <none> and password file <none>
I20260927 11:22:00.887773 20569 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:00.887841 20569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:00.888010 20569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:00.889163 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "2ca401a392fa41f0bcb40943e756c92d"
format_stamp: "Formatted at 2026-09-27 11:22:00 on dist-test-slave-lsvg"
server_key: "dc20da3bf661885b9ed852e80f0c2535"
server_key_iv: "20494b52500ac7b64a59aed494cfbc58"
server_key_version: "encryptionkey@0"
I20260927 11:22:00.889529 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ca401a392fa41f0bcb40943e756c92d"
format_stamp: "Formatted at 2026-09-27 11:22:00 on dist-test-slave-lsvg"
server_key: "dc20da3bf661885b9ed852e80f0c2535"
server_key_iv: "20494b52500ac7b64a59aed494cfbc58"
server_key_version: "encryptionkey@0"
I20260927 11:22:00.890868 20569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:22:00.891631 20584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:00.891804 20569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:22:00.891903 20569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "2ca401a392fa41f0bcb40943e756c92d"
format_stamp: "Formatted at 2026-09-27 11:22:00 on dist-test-slave-lsvg"
server_key: "dc20da3bf661885b9ed852e80f0c2535"
server_key_iv: "20494b52500ac7b64a59aed494cfbc58"
server_key_version: "encryptionkey@0"
I20260927 11:22:00.891986 20569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:00.909266 20569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:00.911100 20569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:00.911263 20569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:00.915424 20569 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46593
I20260927 11:22:00.915460 20636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46593 every 8 connection(s)
I20260927 11:22:00.915817 20569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:00.916699 20637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:00.917853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 20569
I20260927 11:22:00.917960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:00.918190 392 external_mini_cluster.cc:1428] Setting key f60af011dc4ba271b4f278c225260f1f
I20260927 11:22:00.919409 20637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d: Bootstrap starting.
I20260927 11:22:00.920120 20637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:00.920483 20637 log.cc:824] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:00.921283 20637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d: No bootstrap required, opened a new log
I20260927 11:22:00.923462 20637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca401a392fa41f0bcb40943e756c92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260927 11:22:00.923650 20637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:00.923702 20637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca401a392fa41f0bcb40943e756c92d, State: Initialized, Role: FOLLOWER
I20260927 11:22:00.923848 20637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca401a392fa41f0bcb40943e756c92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260927 11:22:00.923923 20637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:00.923949 20637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:00.923992 20637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:00.924707 20637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca401a392fa41f0bcb40943e756c92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260927 11:22:00.924836 20637 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca401a392fa41f0bcb40943e756c92d; no voters:
I20260927 11:22:00.925019 20637 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:00.925112 20642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:00.925273 20642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [term 1 LEADER]: Becoming Leader. State: Replica: 2ca401a392fa41f0bcb40943e756c92d, State: Running, Role: LEADER
I20260927 11:22:00.925316 20637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:00.925359 20642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca401a392fa41f0bcb40943e756c92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260927 11:22:00.925670 20642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca401a392fa41f0bcb40943e756c92d. Latest consensus state: current_term: 1 leader_uuid: "2ca401a392fa41f0bcb40943e756c92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca401a392fa41f0bcb40943e756c92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260927 11:22:00.925765 20642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:00.925686 20643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ca401a392fa41f0bcb40943e756c92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca401a392fa41f0bcb40943e756c92d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260927 11:22:00.925856 20643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d [sys.catalog]: This master's current role is: LEADER
Sep 27 11:22:00 dist-test-slave-lsvg krb5kdc[20556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 11:22:00.926831 20648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:00.927449 20648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:00.929147 20648 catalog_manager.cc:1347] Generated new cluster ID: 02c7ca2eb9fe4f689b29da744c9fafb5
I20260927 11:22:00.929198 20648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:00.930181 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:00.918826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47177 (local address 127.0.0.1:46593)
0927 11:22:00.919027 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:00.919035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:00.919062 (+ 27us) server_negotiation.cc:407] Connection header received
0927 11:22:00.919570 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:00.919583 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:00.919636 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:00.919766 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:00.920613 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:00.921767 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:00.922875 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:00.923311 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:00.923407 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:00.926677 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:00.926682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:00.926682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:00.926696 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:00.928399 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:00.928868 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:00.928871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:00.928872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:00.928984 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:00.929259 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:00.929260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:00.929260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:00.929549 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:00.929720 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:00.929886 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 11:22:00.929990 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260927 11:22:00.931308 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:00.944358 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:00.932802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39657 (local address 127.0.0.1:46593)
0927 11:22:00.932840 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:00.932841 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:00.937246 (+ 4405us) server_negotiation.cc:407] Connection header received
0927 11:22:00.937313 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:00.937314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:00.937324 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:00.937372 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:00.938015 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:00.938760 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:00.939767 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:00.940079 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:00.940138 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:00.941624 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:00.941625 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:00.941625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:00.941631 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:00.943163 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:00.943573 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:00.943574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:00.943574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:00.943627 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:00.943850 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:00.943851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:00.943851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:00.943971 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:00.944060 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:00.944209 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 11:22:00.944267 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:22:00.953370 20648 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:00.954018 20648 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:00.962954 20648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ca401a392fa41f0bcb40943e756c92d: Generated new TSK 0
I20260927 11:22:00.963176 20648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:00.971320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 20569
W20260927 11:22:00.976068 20554 connection.cc:535] client connection to 127.0.0.1:46593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46593): Connection reset by peer (error 104)
2026-09-27T11:22:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:01 dist-test-slave-lsvg krb5kdc[20668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:01 dist-test-slave-lsvg krb5kdc[20668](info): set up 2 sockets
Sep 27 11:22:01 dist-test-slave-lsvg krb5kdc[20668](info): commencing operation
krb5kdc: starting...
W20260927 11:22:02.005252 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:02 dist-test-slave-lsvg krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:22:02.129900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:02.210721 20681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:02.210939 20681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:02.210963 20681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:02.212206 20681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:02.212255 20681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:02.212268 20681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:02.212282 20681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:02.212294 20681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:02.213739 20681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:02.213935 20681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:02.214185 20681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:02.219177 20689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:02.219312 20681 server_base.cc:1034] running on GCE node
W20260927 11:22:02.219254 20686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:02.219178 20687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:02.219713 20681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:02.220033 20681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:02.221232 20681 hybrid_clock.cc:648] HybridClock initialized: now 1790508122221228 us; error 41 us; skew 500 ppm
Sep 27 11:22:02 dist-test-slave-lsvg krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:02.229239 20681 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:22:02.229770 20681 webserver.cc:460] Webserver started at http://127.0.0.1:43197/ using document root <none> and password file <none>
I20260927 11:22:02.230007 20681 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:02.230077 20681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:02.230234 20681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:02.231350 20681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "827982e423784c3f82d18dd9f7a995f5"
format_stamp: "Formatted at 2026-09-27 11:22:02 on dist-test-slave-lsvg"
server_key: "86a20d379ad46f138e87d6f1182dd7c1"
server_key_iv: "7d63b34c8112c9837db7f31b9a297b72"
server_key_version: "encryptionkey@0"
I20260927 11:22:02.231724 20681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "827982e423784c3f82d18dd9f7a995f5"
format_stamp: "Formatted at 2026-09-27 11:22:02 on dist-test-slave-lsvg"
server_key: "86a20d379ad46f138e87d6f1182dd7c1"
server_key_iv: "7d63b34c8112c9837db7f31b9a297b72"
server_key_version: "encryptionkey@0"
I20260927 11:22:02.233170 20681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:22:02.233975 20696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:02.234165 20681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:22:02.234267 20681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "827982e423784c3f82d18dd9f7a995f5"
format_stamp: "Formatted at 2026-09-27 11:22:02 on dist-test-slave-lsvg"
server_key: "86a20d379ad46f138e87d6f1182dd7c1"
server_key_iv: "7d63b34c8112c9837db7f31b9a297b72"
server_key_version: "encryptionkey@0"
I20260927 11:22:02.234356 20681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:02.255628 20681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:02.257702 20681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:02.257876 20681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:02.262133 20681 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32793
I20260927 11:22:02.262169 20748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32793 every 8 connection(s)
I20260927 11:22:02.262566 20681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:02.263450 20749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:02.264937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 20681
I20260927 11:22:02.265107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:02.265285 392 external_mini_cluster.cc:1428] Setting key ac88271db0fe4539a4adfcdb3207fdeb
I20260927 11:22:02.266163 20749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5: Bootstrap starting.
I20260927 11:22:02.266849 20749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:02.267161 20749 log.cc:824] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:02.267879 20749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5: No bootstrap required, opened a new log
I20260927 11:22:02.270310 20749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827982e423784c3f82d18dd9f7a995f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32793 } }
I20260927 11:22:02.270494 20749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:02.270553 20749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827982e423784c3f82d18dd9f7a995f5, State: Initialized, Role: FOLLOWER
I20260927 11:22:02.270712 20749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827982e423784c3f82d18dd9f7a995f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32793 } }
I20260927 11:22:02.270788 20749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:02.270810 20749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:02.270870 20749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:02.271373 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:02.265855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38133 (local address 127.0.0.1:32793)
0927 11:22:02.266036 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:02.266044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:02.266070 (+ 26us) server_negotiation.cc:407] Connection header received
0927 11:22:02.266512 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:02.266520 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:02.266569 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:02.266707 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:02.266991 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:02.268147 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:02.269273 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:02.269678 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:02.269761 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:02.270571 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:02.270611 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:02.270612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:02.270874 (+ 262us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:02.270916 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:02.270932 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:02.271016 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:02.271117 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 11:22:02.271163 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260927 11:22:02.271548 20749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827982e423784c3f82d18dd9f7a995f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32793 } }
I20260927 11:22:02.271656 20749 leader_election.cc:302] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 827982e423784c3f82d18dd9f7a995f5; no voters:
I20260927 11:22:02.271814 20749 leader_election.cc:288] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:02.271881 20754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:02.272080 20749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:02.272102 20754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [term 1 LEADER]: Becoming Leader. State: Replica: 827982e423784c3f82d18dd9f7a995f5, State: Running, Role: LEADER
I20260927 11:22:02.272269 20754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827982e423784c3f82d18dd9f7a995f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32793 } }
I20260927 11:22:02.272336 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:02.272629 20754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827982e423784c3f82d18dd9f7a995f5. Latest consensus state: current_term: 1 leader_uuid: "827982e423784c3f82d18dd9f7a995f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827982e423784c3f82d18dd9f7a995f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32793 } } }
I20260927 11:22:02.272718 20754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:02.272639 20756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "827982e423784c3f82d18dd9f7a995f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827982e423784c3f82d18dd9f7a995f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32793 } } }
I20260927 11:22:02.272835 20756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:02.273136 20763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:02.273665 20763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:02.275368 20763 catalog_manager.cc:1347] Generated new cluster ID: 9b590843ed6f44b28aca13efe12f97c6
I20260927 11:22:02.275418 20763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:02.277360 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:02.273885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45691 (local address 127.0.0.1:32793)
0927 11:22:02.273917 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:02.273919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:02.274092 (+ 173us) server_negotiation.cc:407] Connection header received
0927 11:22:02.274159 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:02.274159 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:02.274167 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:02.274238 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:02.274409 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:02.274991 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:02.275983 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:02.276277 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:02.276345 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:02.277098 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:02.277099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:02.277100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:02.277104 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:02.277118 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:02.277123 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:02.277177 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:02.277280 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 11:22:02.277289 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260927 11:22:02.289306 20763 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:02.290055 20763 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:02.294567 20763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 827982e423784c3f82d18dd9f7a995f5: Generated new TSK 0
I20260927 11:22:02.294737 20763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:02.298679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 20681
W20260927 11:22:02.303112 20666 connection.cc:535] client connection to 127.0.0.1:32793 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32793): Connection reset by peer (error 104)
2026-09-27T11:22:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:02 dist-test-slave-lsvg krb5kdc[20780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:02 dist-test-slave-lsvg krb5kdc[20780](info): set up 2 sockets
Sep 27 11:22:02 dist-test-slave-lsvg krb5kdc[20780](info): commencing operation
krb5kdc: starting...
W20260927 11:22:03.329200 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:03 dist-test-slave-lsvg krb5kdc[20780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:22:03.448796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:03.529883 20793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:03.530143 20793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:03.530180 20793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:03.531347 20793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:03.531414 20793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:03.531440 20793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:03.531461 20793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:03.531489 20793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:03.532961 20793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:03.533270 20793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:03.533497 20793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:22:03.538352 20793 server_base.cc:1034] running on GCE node
W20260927 11:22:03.538321 20799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:03.538471 20798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:03.538314 20801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:03.538785 20793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:03.539081 20793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:03.540244 20793 hybrid_clock.cc:648] HybridClock initialized: now 1790508123540221 us; error 55 us; skew 500 ppm
Sep 27 11:22:03 dist-test-slave-lsvg krb5kdc[20780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:03.542671 20793 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:22:03.543150 20793 webserver.cc:460] Webserver started at http://127.0.0.1:38541/ using document root <none> and password file <none>
I20260927 11:22:03.543368 20793 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:03.543443 20793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:03.543591 20793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:03.544659 20793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "9bfcb834e2b14e509493ed83c6252e88"
format_stamp: "Formatted at 2026-09-27 11:22:03 on dist-test-slave-lsvg"
server_key: "8874332a7a0ab69cbd11cf059f6f93b1"
server_key_iv: "69dfca6886541685a34ce06b3caa8bd5"
server_key_version: "encryptionkey@0"
I20260927 11:22:03.545094 20793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "9bfcb834e2b14e509493ed83c6252e88"
format_stamp: "Formatted at 2026-09-27 11:22:03 on dist-test-slave-lsvg"
server_key: "8874332a7a0ab69cbd11cf059f6f93b1"
server_key_iv: "69dfca6886541685a34ce06b3caa8bd5"
server_key_version: "encryptionkey@0"
I20260927 11:22:03.546599 20793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:22:03.547429 20808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:03.547641 20793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:22:03.547744 20793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "9bfcb834e2b14e509493ed83c6252e88"
format_stamp: "Formatted at 2026-09-27 11:22:03 on dist-test-slave-lsvg"
server_key: "8874332a7a0ab69cbd11cf059f6f93b1"
server_key_iv: "69dfca6886541685a34ce06b3caa8bd5"
server_key_version: "encryptionkey@0"
I20260927 11:22:03.547833 20793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:03.551194 20793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:03.553088 20793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:03.553232 20793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:03.556893 20793 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37733
I20260927 11:22:03.557026 20860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37733 every 8 connection(s)
I20260927 11:22:03.557324 20793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:03.558333 20861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:03.560575 20861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88: Bootstrap starting.
I20260927 11:22:03.561219 20861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:03.561514 20861 log.cc:824] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:03.562220 20861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88: No bootstrap required, opened a new log
I20260927 11:22:03.563356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 20793
I20260927 11:22:03.563524 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:03.563812 392 external_mini_cluster.cc:1428] Setting key a25e190050209cb6973be52fb545b99b
I20260927 11:22:03.563974 20861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfcb834e2b14e509493ed83c6252e88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37733 } }
I20260927 11:22:03.564189 20861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:03.564236 20861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfcb834e2b14e509493ed83c6252e88, State: Initialized, Role: FOLLOWER
I20260927 11:22:03.564432 20861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfcb834e2b14e509493ed83c6252e88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37733 } }
I20260927 11:22:03.564513 20861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:03.564538 20861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:03.564603 20861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:03.565737 20861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfcb834e2b14e509493ed83c6252e88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37733 } }
I20260927 11:22:03.565864 20861 leader_election.cc:302] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bfcb834e2b14e509493ed83c6252e88; no voters:
I20260927 11:22:03.566025 20861 leader_election.cc:288] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:03.566089 20866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:03.566268 20866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [term 1 LEADER]: Becoming Leader. State: Replica: 9bfcb834e2b14e509493ed83c6252e88, State: Running, Role: LEADER
I20260927 11:22:03.566359 20861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:03.566428 20866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfcb834e2b14e509493ed83c6252e88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37733 } }
I20260927 11:22:03.566783 20866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bfcb834e2b14e509493ed83c6252e88. Latest consensus state: current_term: 1 leader_uuid: "9bfcb834e2b14e509493ed83c6252e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfcb834e2b14e509493ed83c6252e88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37733 } } }
I20260927 11:22:03.566793 20867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bfcb834e2b14e509493ed83c6252e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfcb834e2b14e509493ed83c6252e88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37733 } } }
I20260927 11:22:03.567041 20867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:03.567338 20866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:03.567430 20875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:03.566365 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:03.564431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37721 (local address 127.0.0.1:37733)
0927 11:22:03.564702 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:03.564712 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:03.564772 (+ 60us) server_negotiation.cc:407] Connection header received
0927 11:22:03.564938 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:03.564945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:03.564991 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:03.565156 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:03.565821 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:03.565825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:03.565826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:03.565831 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:03.565881 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:03.565896 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:03.565968 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:03.566097 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 11:22:03.566144 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":118,"threads_started":1}
I20260927 11:22:03.567934 20875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:03.568594 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:03.569855 20875 catalog_manager.cc:1347] Generated new cluster ID: fddcf33dd5964f6d8f4a64429df6cdcc
I20260927 11:22:03.569921 20875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:03.574348 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:03.570380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59773 (local address 127.0.0.1:37733)
0927 11:22:03.570420 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:03.570421 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:03.573372 (+ 2951us) server_negotiation.cc:407] Connection header received
0927 11:22:03.573434 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:03.573435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:03.573450 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:03.573518 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:03.574156 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:03.574157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:03.574157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:03.574162 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:03.574176 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:03.574180 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:03.574232 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:03.574275 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 11:22:03.574287 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:22:03.598313 20875 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:03.598503 20875 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:03.607003 20875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bfcb834e2b14e509493ed83c6252e88: Generated new TSK 0
I20260927 11:22:03.607221 20875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:03.612284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 20793
2026-09-27T11:22:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:03 dist-test-slave-lsvg krb5kdc[20892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:03 dist-test-slave-lsvg krb5kdc[20892](info): set up 2 sockets
Sep 27 11:22:03 dist-test-slave-lsvg krb5kdc[20892](info): commencing operation
krb5kdc: starting...
W20260927 11:22:04.657554 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:04 dist-test-slave-lsvg krb5kdc[20892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:22:04.775995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:04.857156 20905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:04.857425 20905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:04.857477 20905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:04.858668 20905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:04.858745 20905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:04.858772 20905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:04.858800 20905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:04.858831 20905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:04.860329 20905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:04.860656 20905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:04.860900 20905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:22:04.866101 20905 server_base.cc:1034] running on GCE node
W20260927 11:22:04.866081 20910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:04.866081 20913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:04.866060 20911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:04.866577 20905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:04.866905 20905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:04.868079 20905 hybrid_clock.cc:648] HybridClock initialized: now 1790508124868065 us; error 34 us; skew 500 ppm
Sep 27 11:22:04 dist-test-slave-lsvg krb5kdc[20892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:04.870929 20905 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:22:04.871428 20905 webserver.cc:460] Webserver started at http://127.0.0.1:36459/ using document root <none> and password file <none>
I20260927 11:22:04.871639 20905 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:04.871683 20905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:04.871774 20905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:04.872807 20905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "aad2ce86c34144c99422af76b0c857e3"
format_stamp: "Formatted at 2026-09-27 11:22:04 on dist-test-slave-lsvg"
server_key: "6d72a47f7f639b9916fd3017ae9828f5"
server_key_iv: "2bb7da4ae02ed14369061778ff292051"
server_key_version: "encryptionkey@0"
I20260927 11:22:04.873188 20905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "aad2ce86c34144c99422af76b0c857e3"
format_stamp: "Formatted at 2026-09-27 11:22:04 on dist-test-slave-lsvg"
server_key: "6d72a47f7f639b9916fd3017ae9828f5"
server_key_iv: "2bb7da4ae02ed14369061778ff292051"
server_key_version: "encryptionkey@0"
I20260927 11:22:04.874572 20905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:22:04.875238 20920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:04.875416 20905 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:22:04.875523 20905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "aad2ce86c34144c99422af76b0c857e3"
format_stamp: "Formatted at 2026-09-27 11:22:04 on dist-test-slave-lsvg"
server_key: "6d72a47f7f639b9916fd3017ae9828f5"
server_key_iv: "2bb7da4ae02ed14369061778ff292051"
server_key_version: "encryptionkey@0"
I20260927 11:22:04.875607 20905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:04.888324 20905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:04.890259 20905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:04.890429 20905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:04.894583 20905 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45739
I20260927 11:22:04.894623 20972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45739 every 8 connection(s)
I20260927 11:22:04.895006 20905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:04.895869 20973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:04.898227 20973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3: Bootstrap starting.
I20260927 11:22:04.898756 20973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:04.898983 20973 log.cc:824] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:04.899618 20973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3: No bootstrap required, opened a new log
I20260927 11:22:04.900911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 20905
I20260927 11:22:04.901062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:04.901255 392 external_mini_cluster.cc:1428] Setting key 47588e555549b1b33cd71a3d84b202df
I20260927 11:22:04.901539 20973 raft_consensus.cc:348] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad2ce86c34144c99422af76b0c857e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } }
I20260927 11:22:04.901782 20973 raft_consensus.cc:374] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:04.901844 20973 raft_consensus.cc:729] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad2ce86c34144c99422af76b0c857e3, State: Initialized, Role: FOLLOWER
I20260927 11:22:04.902045 20973 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad2ce86c34144c99422af76b0c857e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } }
I20260927 11:22:04.902153 20973 raft_consensus.cc:388] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:04.902197 20973 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:04.902252 20973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:04.903110 20973 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad2ce86c34144c99422af76b0c857e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } }
I20260927 11:22:04.903231 20973 leader_election.cc:302] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aad2ce86c34144c99422af76b0c857e3; no voters:
I20260927 11:22:04.903429 20973 leader_election.cc:288] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:04.903457 20978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:04.903561 20978 raft_consensus.cc:686] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [term 1 LEADER]: Becoming Leader. State: Replica: aad2ce86c34144c99422af76b0c857e3, State: Running, Role: LEADER
I20260927 11:22:04.903699 20973 sys_catalog.cc:564] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:04.903707 20978 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad2ce86c34144c99422af76b0c857e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } }
I20260927 11:22:04.904096 20979 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aad2ce86c34144c99422af76b0c857e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad2ce86c34144c99422af76b0c857e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } } }
I20260927 11:22:04.904191 20979 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:04.904116 20980 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad2ce86c34144c99422af76b0c857e3. Latest consensus state: current_term: 1 leader_uuid: "aad2ce86c34144c99422af76b0c857e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad2ce86c34144c99422af76b0c857e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } } }
I20260927 11:22:04.904337 20980 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3 [sys.catalog]: This master's current role is: LEADER
W20260927 11:22:04.905659 20994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:22:04.905717 20994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:22:04.905767 20985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:04.905918 20985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:04.907510 20985 catalog_manager.cc:1347] Generated new cluster ID: e864b6a94aa948a7ad5fde22b7a303fd
I20260927 11:22:04.907564 20985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:22:04 dist-test-slave-lsvg krb5kdc[20892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 11:22:04.912163 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:04.901788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49241 (local address 127.0.0.1:45739)
0927 11:22:04.901948 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:04.901964 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:04.901992 (+ 28us) server_negotiation.cc:407] Connection header received
0927 11:22:04.902135 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:04.902141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:04.902206 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:04.902353 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:04.903007 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:04.904320 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:04.905360 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:04.905966 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:04.906080 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:04.908869 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:04.908873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:04.908873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:04.908887 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:04.910640 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:04.911018 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:04.911021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:04.911022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:04.911121 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:04.911328 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:04.911329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:04.911330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:04.911567 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:04.911732 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:04.911825 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 11:22:04.911966 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260927 11:22:04.913106 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:04.928283 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:04.914419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48101 (local address 127.0.0.1:45739)
0927 11:22:04.914456 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:04.914458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:04.921221 (+ 6763us) server_negotiation.cc:407] Connection header received
0927 11:22:04.921297 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:04.921298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:04.921309 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:04.921352 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:04.922974 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:04.923478 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:04.924189 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:04.924386 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:04.924472 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:04.925735 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:04.925737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:04.925737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:04.925743 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:04.927157 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:04.927516 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:04.927517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:04.927517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:04.927576 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:04.927772 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:04.927773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:04.927773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:04.927885 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:04.928013 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:04.928098 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 11:22:04.928173 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:22:04.929271 20985 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:04.929767 20985 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:04.940279 20985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aad2ce86c34144c99422af76b0c857e3: Generated new TSK 0
I20260927 11:22:04.940491 20985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:04.951146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 20905
2026-09-27T11:22:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:05 dist-test-slave-lsvg krb5kdc[21004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:05 dist-test-slave-lsvg krb5kdc[21004](info): set up 2 sockets
Sep 27 11:22:05 dist-test-slave-lsvg krb5kdc[21004](info): commencing operation
krb5kdc: starting...
W20260927 11:22:05.981164 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:06 dist-test-slave-lsvg krb5kdc[21004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:22:06.100988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:06.181255 21017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:06.181463 21017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:06.181485 21017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:06.182570 21017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:06.182605 21017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:06.182619 21017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:06.182632 21017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:06.182644 21017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:06.184031 21017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:06.184219 21017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:06.184388 21017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:06.189505 21025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:06.189561 21022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:06.189529 21023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:06.189636 21017 server_base.cc:1034] running on GCE node
I20260927 11:22:06.190001 21017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:06.190311 21017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:06.191498 21017 hybrid_clock.cc:648] HybridClock initialized: now 1790508126191442 us; error 65 us; skew 500 ppm
Sep 27 11:22:06 dist-test-slave-lsvg krb5kdc[21004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:06.194057 21017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:22:06.194536 21017 webserver.cc:460] Webserver started at http://127.0.0.1:39837/ using document root <none> and password file <none>
I20260927 11:22:06.194763 21017 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:06.194832 21017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:06.194993 21017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:06.196049 21017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "2b485e22b13a47299a50d15c09a692bf"
format_stamp: "Formatted at 2026-09-27 11:22:06 on dist-test-slave-lsvg"
server_key: "14bde7c393b3c23fc9582685040623e3"
server_key_iv: "15688f73eabd51b5a7abfc4bfa680476"
server_key_version: "encryptionkey@0"
I20260927 11:22:06.196417 21017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b485e22b13a47299a50d15c09a692bf"
format_stamp: "Formatted at 2026-09-27 11:22:06 on dist-test-slave-lsvg"
server_key: "14bde7c393b3c23fc9582685040623e3"
server_key_iv: "15688f73eabd51b5a7abfc4bfa680476"
server_key_version: "encryptionkey@0"
I20260927 11:22:06.197908 21017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:22:06.198750 21032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:06.198966 21017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:22:06.199083 21017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "2b485e22b13a47299a50d15c09a692bf"
format_stamp: "Formatted at 2026-09-27 11:22:06 on dist-test-slave-lsvg"
server_key: "14bde7c393b3c23fc9582685040623e3"
server_key_iv: "15688f73eabd51b5a7abfc4bfa680476"
server_key_version: "encryptionkey@0"
I20260927 11:22:06.199159 21017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:06.230157 21017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:06.232658 21017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:06.232829 21017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:06.237107 21017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38169
I20260927 11:22:06.237151 21084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38169 every 8 connection(s)
I20260927 11:22:06.237567 21017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:06.238423 21085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:06.240669 21085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf: Bootstrap starting.
I20260927 11:22:06.241286 21085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:06.241569 21085 log.cc:824] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:06.242230 21085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf: No bootstrap required, opened a new log
I20260927 11:22:06.243790 21085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b485e22b13a47299a50d15c09a692bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260927 11:22:06.243925 21085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:06.243947 21085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b485e22b13a47299a50d15c09a692bf, State: Initialized, Role: FOLLOWER
I20260927 11:22:06.244120 21085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b485e22b13a47299a50d15c09a692bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260927 11:22:06.244182 21085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:06.244204 21085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:06.244268 21085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:06.244808 21085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b485e22b13a47299a50d15c09a692bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260927 11:22:06.244895 21085 leader_election.cc:302] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b485e22b13a47299a50d15c09a692bf; no voters:
I20260927 11:22:06.245131 21085 leader_election.cc:288] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:06.245214 21088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:06.245378 21088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [term 1 LEADER]: Becoming Leader. State: Replica: 2b485e22b13a47299a50d15c09a692bf, State: Running, Role: LEADER
I20260927 11:22:06.245395 21085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:06.245466 21088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b485e22b13a47299a50d15c09a692bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } }
I20260927 11:22:06.245867 21090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b485e22b13a47299a50d15c09a692bf. Latest consensus state: current_term: 1 leader_uuid: "2b485e22b13a47299a50d15c09a692bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b485e22b13a47299a50d15c09a692bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } } }
I20260927 11:22:06.245862 21089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b485e22b13a47299a50d15c09a692bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b485e22b13a47299a50d15c09a692bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38169 } } }
I20260927 11:22:06.245958 21090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:06.245965 21089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:06.246146 21092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:06.246315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 21017
I20260927 11:22:06.246445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:06.246662 392 external_mini_cluster.cc:1428] Setting key 3e97cde9b999e815e3720caf2e2c09c9
I20260927 11:22:06.246702 21092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:06.248274 21092 catalog_manager.cc:1347] Generated new cluster ID: d826ce2fdd6c451b8c3145068c8f52d5
I20260927 11:22:06.248314 21092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:06.254522 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:06.247170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49273 (local address 127.0.0.1:38169)
0927 11:22:06.247335 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:06.247342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:06.249284 (+ 1942us) server_negotiation.cc:407] Connection header received
0927 11:22:06.249406 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:06.249413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:06.249467 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:06.249606 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:06.249833 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:06.251202 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:06.252331 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:06.252936 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:06.253032 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:06.253841 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:06.253845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:06.253846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:06.254051 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:06.254087 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:06.254100 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:06.254157 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:06.254256 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 11:22:06.254317 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260927 11:22:06.255450 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:06.260151 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:06.256838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42753 (local address 127.0.0.1:38169)
0927 11:22:06.256875 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:06.256877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:06.256938 (+ 61us) server_negotiation.cc:407] Connection header received
0927 11:22:06.257020 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:06.257021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:06.257030 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:06.257075 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:06.257240 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:06.257940 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:06.258891 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:06.259185 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:06.259221 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:06.259917 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:06.259918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:06.259918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:06.259922 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:06.259935 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:06.259939 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:06.259963 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:06.260055 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 11:22:06.260078 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:22:06.261788 21092 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:06.262475 21092 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:06.276142 21092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b485e22b13a47299a50d15c09a692bf: Generated new TSK 0
I20260927 11:22:06.276347 21092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:06.282533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 21017
2026-09-27T11:22:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:06 dist-test-slave-lsvg krb5kdc[21116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:06 dist-test-slave-lsvg krb5kdc[21116](info): set up 2 sockets
Sep 27 11:22:06 dist-test-slave-lsvg krb5kdc[21116](info): commencing operation
krb5kdc: starting...
W20260927 11:22:07.353492 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:07 dist-test-slave-lsvg krb5kdc[21116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:22:07.471798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:07.553617 21129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:07.553836 21129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:07.553859 21129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:07.554945 21129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:07.554986 21129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:07.555001 21129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:07.555014 21129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:07.555025 21129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:07.556438 21129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:07.556638 21129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:07.556818 21129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:07.561975 21134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:07.562044 21135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:07.562011 21137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:07.562136 21129 server_base.cc:1034] running on GCE node
I20260927 11:22:07.562464 21129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:07.562769 21129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:07.563936 21129 hybrid_clock.cc:648] HybridClock initialized: now 1790508127563915 us; error 44 us; skew 500 ppm
Sep 27 11:22:07 dist-test-slave-lsvg krb5kdc[21116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:07.572422 21129 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:22:07.572891 21129 webserver.cc:460] Webserver started at http://127.0.0.1:36357/ using document root <none> and password file <none>
I20260927 11:22:07.573201 21129 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:07.573272 21129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:07.573463 21129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:07.574571 21129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "50004665956e4d61ac3c73f5fa627715"
format_stamp: "Formatted at 2026-09-27 11:22:07 on dist-test-slave-lsvg"
server_key: "bfbedbefd823dc401e939c7a9c4a37d2"
server_key_iv: "50caae2531fab98a50458103eaee90b8"
server_key_version: "encryptionkey@0"
I20260927 11:22:07.574936 21129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "50004665956e4d61ac3c73f5fa627715"
format_stamp: "Formatted at 2026-09-27 11:22:07 on dist-test-slave-lsvg"
server_key: "bfbedbefd823dc401e939c7a9c4a37d2"
server_key_iv: "50caae2531fab98a50458103eaee90b8"
server_key_version: "encryptionkey@0"
I20260927 11:22:07.576272 21129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:22:07.577154 21144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:07.577311 21129 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:22:07.577443 21129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "50004665956e4d61ac3c73f5fa627715"
format_stamp: "Formatted at 2026-09-27 11:22:07 on dist-test-slave-lsvg"
server_key: "bfbedbefd823dc401e939c7a9c4a37d2"
server_key_iv: "50caae2531fab98a50458103eaee90b8"
server_key_version: "encryptionkey@0"
I20260927 11:22:07.577526 21129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:07.580775 21129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:07.582677 21129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:07.582832 21129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:07.586942 21129 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46069
I20260927 11:22:07.587054 21196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46069 every 8 connection(s)
I20260927 11:22:07.587375 21129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:07.588369 21197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:07.590790 21197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715: Bootstrap starting.
I20260927 11:22:07.591374 21197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:07.591675 21197 log.cc:824] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:07.592429 21197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715: No bootstrap required, opened a new log
I20260927 11:22:07.594156 21197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50004665956e4d61ac3c73f5fa627715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } }
I20260927 11:22:07.594296 21197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:07.594319 21197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50004665956e4d61ac3c73f5fa627715, State: Initialized, Role: FOLLOWER
I20260927 11:22:07.594517 21197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50004665956e4d61ac3c73f5fa627715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } }
I20260927 11:22:07.594583 21197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:07.594604 21197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:07.594671 21197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:07.595232 21197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50004665956e4d61ac3c73f5fa627715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } }
I20260927 11:22:07.595345 21197 leader_election.cc:302] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50004665956e4d61ac3c73f5fa627715; no voters:
I20260927 11:22:07.595536 21197 leader_election.cc:288] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:07.595607 21200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:07.595773 21200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [term 1 LEADER]: Becoming Leader. State: Replica: 50004665956e4d61ac3c73f5fa627715, State: Running, Role: LEADER
I20260927 11:22:07.595824 21197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:07.595942 21200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50004665956e4d61ac3c73f5fa627715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } }
I20260927 11:22:07.596335 21202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50004665956e4d61ac3c73f5fa627715. Latest consensus state: current_term: 1 leader_uuid: "50004665956e4d61ac3c73f5fa627715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50004665956e4d61ac3c73f5fa627715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } } }
I20260927 11:22:07.596426 21202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:07.596324 21201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50004665956e4d61ac3c73f5fa627715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50004665956e4d61ac3c73f5fa627715" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46069 } } }
I20260927 11:22:07.596570 21201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:07.597026 21204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:07.597162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 21129
I20260927 11:22:07.597290 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:07.597493 392 external_mini_cluster.cc:1428] Setting key 9594f1c5f209f66a34b9b650b6601df8
I20260927 11:22:07.597630 21204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:07.599922 21204 catalog_manager.cc:1347] Generated new cluster ID: 25bbd9480f5748f8aee7d0daf711bfb4
I20260927 11:22:07.599978 21204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:07.600247 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:07.598212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53817 (local address 127.0.0.1:46069)
0927 11:22:07.598380 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:07.598391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:07.598423 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:22:07.598688 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:07.598696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:07.598727 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:07.598873 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:07.599690 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:07.599693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:07.599694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:07.599698 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:07.599737 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:07.599752 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:07.599822 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:07.600037 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 11:22:07.600078 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260927 11:22:07.601162 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:07.603830 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:07.602734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39615 (local address 127.0.0.1:46069)
0927 11:22:07.602769 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:07.602771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:07.602922 (+ 151us) server_negotiation.cc:407] Connection header received
0927 11:22:07.602974 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:07.602975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:07.602980 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:07.603044 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:07.603606 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:07.603607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:07.603608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:07.603612 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:07.603626 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:07.603631 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:07.603692 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:07.603754 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 11:22:07.603766 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:22:07.627007 21204 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:07.627225 21204 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:07.630002 21204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50004665956e4d61ac3c73f5fa627715: Generated new TSK 0
I20260927 11:22:07.630214 21204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:07.639259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 21129
2026-09-27T11:22:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:07 dist-test-slave-lsvg krb5kdc[21228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:07 dist-test-slave-lsvg krb5kdc[21228](info): set up 2 sockets
Sep 27 11:22:07 dist-test-slave-lsvg krb5kdc[21228](info): commencing operation
krb5kdc: starting...
W20260927 11:22:08.669183 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:08 dist-test-slave-lsvg krb5kdc[21228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:22:08.788275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:08.869730 21241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:08.869961 21241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:08.869983 21241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:08.871272 21241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:08.871321 21241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:08.871336 21241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:08.871373 21241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:08.871400 21241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:08.872948 21241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:08.873236 21241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:08.873492 21241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:08.878597 21249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:08.878741 21241 server_base.cc:1034] running on GCE node
W20260927 11:22:08.878620 21247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:08.878834 21246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:08.879149 21241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:08.879419 21241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:08.880577 21241 hybrid_clock.cc:648] HybridClock initialized: now 1790508128880550 us; error 42 us; skew 500 ppm
Sep 27 11:22:08 dist-test-slave-lsvg krb5kdc[21228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:08.883364 21241 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:22:08.883875 21241 webserver.cc:460] Webserver started at http://127.0.0.1:34419/ using document root <none> and password file <none>
I20260927 11:22:08.884135 21241 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:08.884208 21241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:08.884359 21241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:08.885537 21241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "7f54cc0ca3214b648e0a6f31850eccd4"
format_stamp: "Formatted at 2026-09-27 11:22:08 on dist-test-slave-lsvg"
server_key: "62a7d6ed84fde4ee6b42a97518d86f2d"
server_key_iv: "64d55b70a82e1bc01cc05d47bf334c6e"
server_key_version: "encryptionkey@0"
I20260927 11:22:08.885905 21241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "7f54cc0ca3214b648e0a6f31850eccd4"
format_stamp: "Formatted at 2026-09-27 11:22:08 on dist-test-slave-lsvg"
server_key: "62a7d6ed84fde4ee6b42a97518d86f2d"
server_key_iv: "64d55b70a82e1bc01cc05d47bf334c6e"
server_key_version: "encryptionkey@0"
I20260927 11:22:08.887321 21241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:22:08.888173 21256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:08.888386 21241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:22:08.888496 21241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "7f54cc0ca3214b648e0a6f31850eccd4"
format_stamp: "Formatted at 2026-09-27 11:22:08 on dist-test-slave-lsvg"
server_key: "62a7d6ed84fde4ee6b42a97518d86f2d"
server_key_iv: "64d55b70a82e1bc01cc05d47bf334c6e"
server_key_version: "encryptionkey@0"
I20260927 11:22:08.888579 21241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:08.908684 21241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:08.911288 21241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:08.911471 21241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:08.915871 21241 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36369
I20260927 11:22:08.915908 21308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36369 every 8 connection(s)
I20260927 11:22:08.916287 21241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:08.917264 21309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:08.919569 21309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4: Bootstrap starting.
I20260927 11:22:08.920159 21309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:08.920439 21309 log.cc:824] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:08.921162 21309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4: No bootstrap required, opened a new log
I20260927 11:22:08.922736 21309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f54cc0ca3214b648e0a6f31850eccd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36369 } }
I20260927 11:22:08.922873 21309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:08.922894 21309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f54cc0ca3214b648e0a6f31850eccd4, State: Initialized, Role: FOLLOWER
I20260927 11:22:08.923105 21309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f54cc0ca3214b648e0a6f31850eccd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36369 } }
I20260927 11:22:08.923161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 21241
I20260927 11:22:08.923210 21309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:08.923249 21309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:08.923285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:08.923301 21309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:08.923467 392 external_mini_cluster.cc:1428] Setting key 488dfcc7aed7cec44168835f32f24507
I20260927 11:22:08.923939 21309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f54cc0ca3214b648e0a6f31850eccd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36369 } }
I20260927 11:22:08.924091 21309 leader_election.cc:302] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f54cc0ca3214b648e0a6f31850eccd4; no voters:
I20260927 11:22:08.924328 21309 leader_election.cc:288] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:08.924355 21314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:08.924470 21314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [term 1 LEADER]: Becoming Leader. State: Replica: 7f54cc0ca3214b648e0a6f31850eccd4, State: Running, Role: LEADER
I20260927 11:22:08.924615 21309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:08.924609 21314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f54cc0ca3214b648e0a6f31850eccd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36369 } }
I20260927 11:22:08.925117 21315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f54cc0ca3214b648e0a6f31850eccd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f54cc0ca3214b648e0a6f31850eccd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36369 } } }
I20260927 11:22:08.925209 21315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:08.925343 21316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f54cc0ca3214b648e0a6f31850eccd4. Latest consensus state: current_term: 1 leader_uuid: "7f54cc0ca3214b648e0a6f31850eccd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f54cc0ca3214b648e0a6f31850eccd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36369 } } }
I20260927 11:22:08.925410 21316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:08.925822 21318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:08.926350 21318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:08.928208 21318 catalog_manager.cc:1347] Generated new cluster ID: a9ea2f1ef0104295994e2ecdf16d7965
I20260927 11:22:08.928259 21318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:22:08 dist-test-slave-lsvg krb5kdc[21228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 11:22:08.934698 21318 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:08.935374 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:08.924003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:48411 (local address 127.0.0.1:36369)
0927 11:22:08.924164 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:08.924176 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:08.924218 (+ 42us) server_negotiation.cc:407] Connection header received
0927 11:22:08.924327 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:08.924332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:08.924398 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:08.925373 (+ 975us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:08.925484 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:08.926912 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:08.927832 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:08.928385 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:08.928494 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:08.931378 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:08.931383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:08.931383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:08.931396 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:08.933569 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:08.934056 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:08.934059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:08.934060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:08.934177 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:08.934509 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:08.934510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:08.934510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:08.934809 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:08.934969 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:08.935105 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 11:22:08.935209 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260927 11:22:08.935529 21318 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:08.936375 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:08.944620 21318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f54cc0ca3214b648e0a6f31850eccd4: Generated new TSK 0
I20260927 11:22:08.944820 21318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:08.947443 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:08.937853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:40103 (local address 127.0.0.1:36369)
0927 11:22:08.937905 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:08.937907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:08.941230 (+ 3323us) server_negotiation.cc:407] Connection header received
0927 11:22:08.941297 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:08.941298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:08.941308 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:08.941352 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:08.941890 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:08.942425 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:08.943182 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:08.943420 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:08.943505 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:08.944863 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:08.944864 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:08.944865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:08.944871 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:08.946258 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:08.946623 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:08.946625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:08.946625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:08.946679 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:08.946920 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:08.946921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:08.946921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:08.947027 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:08.947179 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:08.947282 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 11:22:08.947345 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:22:08.949401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 21241
W20260927 11:22:08.953656 21226 connection.cc:535] client connection to 127.0.0.1:36369 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36369): Connection reset by peer (error 104)
2026-09-27T11:22:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:09 dist-test-slave-lsvg krb5kdc[21340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:09 dist-test-slave-lsvg krb5kdc[21340](info): set up 2 sockets
Sep 27 11:22:09 dist-test-slave-lsvg krb5kdc[21340](info): commencing operation
krb5kdc: starting...
W20260927 11:22:09.989228 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:10 dist-test-slave-lsvg krb5kdc[21340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:22:10.111835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:10.193359 21353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:10.193636 21353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:10.193681 21353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:10.194927 21353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:10.195009 21353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:10.195034 21353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:10.195065 21353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:10.195086 21353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:10.196650 21353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:10.196954 21353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:10.197220 21353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:10.202134 21358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:10.202140 21359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:10.202308 21361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:10.202349 21353 server_base.cc:1034] running on GCE node
I20260927 11:22:10.202674 21353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:10.202988 21353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:10.204159 21353 hybrid_clock.cc:648] HybridClock initialized: now 1790508130204139 us; error 34 us; skew 500 ppm
Sep 27 11:22:10 dist-test-slave-lsvg krb5kdc[21340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:10.206876 21353 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:22:10.207367 21353 webserver.cc:460] Webserver started at http://127.0.0.1:39537/ using document root <none> and password file <none>
I20260927 11:22:10.207629 21353 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:10.207682 21353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:10.207818 21353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:10.208891 21353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "5653599714ed45f6ada725d78e2dbcaa"
format_stamp: "Formatted at 2026-09-27 11:22:10 on dist-test-slave-lsvg"
server_key: "0c2632db21e5046d3b8b73034f29921d"
server_key_iv: "348c1a93592193936bf8182fd396712f"
server_key_version: "encryptionkey@0"
I20260927 11:22:10.209300 21353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "5653599714ed45f6ada725d78e2dbcaa"
format_stamp: "Formatted at 2026-09-27 11:22:10 on dist-test-slave-lsvg"
server_key: "0c2632db21e5046d3b8b73034f29921d"
server_key_iv: "348c1a93592193936bf8182fd396712f"
server_key_version: "encryptionkey@0"
I20260927 11:22:10.210718 21353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:22:10.211572 21368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:10.211820 21353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:22:10.211925 21353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "5653599714ed45f6ada725d78e2dbcaa"
format_stamp: "Formatted at 2026-09-27 11:22:10 on dist-test-slave-lsvg"
server_key: "0c2632db21e5046d3b8b73034f29921d"
server_key_iv: "348c1a93592193936bf8182fd396712f"
server_key_version: "encryptionkey@0"
I20260927 11:22:10.212018 21353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:10.239327 21353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:10.241706 21353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:10.241851 21353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:10.245944 21353 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46761
I20260927 11:22:10.245980 21420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46761 every 8 connection(s)
I20260927 11:22:10.246349 21353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:10.247212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 21353
I20260927 11:22:10.247363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:10.247565 392 external_mini_cluster.cc:1428] Setting key 260c18f10bcf2e4711a159296503b837
I20260927 11:22:10.247681 21421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:10.250697 21421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa: Bootstrap starting.
I20260927 11:22:10.251305 21421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:10.251595 21421 log.cc:824] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:10.252262 21421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa: No bootstrap required, opened a new log
I20260927 11:22:10.253245 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:10.248158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:59095 (local address 127.0.0.1:46761)
0927 11:22:10.248361 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:10.248378 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:10.248404 (+ 26us) server_negotiation.cc:407] Connection header received
0927 11:22:10.248878 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:10.248885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:10.248936 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:10.249111 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:10.249318 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:10.250225 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:10.251090 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:10.251559 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:10.251679 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:10.252498 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:10.252502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:10.252503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:10.252702 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:10.252742 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:10.252758 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:10.252850 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:10.252975 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 11:22:10.253006 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20260927 11:22:10.254123 21421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5653599714ed45f6ada725d78e2dbcaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46761 } }
I20260927 11:22:10.254262 21421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:10.254285 21421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5653599714ed45f6ada725d78e2dbcaa, State: Initialized, Role: FOLLOWER
I20260927 11:22:10.254397 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:10.254436 21421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5653599714ed45f6ada725d78e2dbcaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46761 } }
I20260927 11:22:10.254514 21421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:10.254547 21421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:10.254592 21421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:10.255319 21421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5653599714ed45f6ada725d78e2dbcaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46761 } }
I20260927 11:22:10.255447 21421 leader_election.cc:302] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5653599714ed45f6ada725d78e2dbcaa; no voters:
I20260927 11:22:10.255630 21421 leader_election.cc:288] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:10.255671 21429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:10.255839 21429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [term 1 LEADER]: Becoming Leader. State: Replica: 5653599714ed45f6ada725d78e2dbcaa, State: Running, Role: LEADER
I20260927 11:22:10.255903 21421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:10.256012 21429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5653599714ed45f6ada725d78e2dbcaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46761 } }
I20260927 11:22:10.256400 21431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5653599714ed45f6ada725d78e2dbcaa. Latest consensus state: current_term: 1 leader_uuid: "5653599714ed45f6ada725d78e2dbcaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5653599714ed45f6ada725d78e2dbcaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46761 } } }
I20260927 11:22:10.256536 21431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:10.256542 21432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5653599714ed45f6ada725d78e2dbcaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5653599714ed45f6ada725d78e2dbcaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46761 } } }
I20260927 11:22:10.256714 21432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:10.256894 21437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:10.257447 21437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:10.259770 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:10.255938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:41801 (local address 127.0.0.1:46761)
0927 11:22:10.255977 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:10.255980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:10.256318 (+ 338us) server_negotiation.cc:407] Connection header received
0927 11:22:10.256408 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:10.256409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:10.256420 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:10.256486 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:10.256677 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:10.257505 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:10.258553 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:10.258748 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:10.258807 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:10.259469 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:10.259470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:10.259470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:10.259474 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:10.259487 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:10.259491 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:10.259555 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:10.259651 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 11:22:10.259661 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:22:10.262137 21437 catalog_manager.cc:1347] Generated new cluster ID: f4b72526decb494e9f336cac6155eb71
I20260927 11:22:10.262192 21437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:10.279995 21437 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:10.280620 21437 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:10.282985 21437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5653599714ed45f6ada725d78e2dbcaa: Generated new TSK 0
I20260927 11:22:10.283207 21437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:10.295574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 21353
W20260927 11:22:10.300010 21338 connection.cc:535] client connection to 127.0.0.1:46761 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46761): Connection reset by peer (error 104)
2026-09-27T11:22:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:10 dist-test-slave-lsvg krb5kdc[21452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:10 dist-test-slave-lsvg krb5kdc[21452](info): set up 2 sockets
Sep 27 11:22:10 dist-test-slave-lsvg krb5kdc[21452](info): commencing operation
krb5kdc: starting...
W20260927 11:22:11.337426 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:11 dist-test-slave-lsvg krb5kdc[21452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:22:11.458288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:11.540427 21465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:11.540647 21465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:11.540670 21465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:11.541805 21465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:11.541841 21465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:11.541855 21465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:11.541868 21465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:11.541880 21465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:11.543310 21465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:11.543517 21465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:11.543684 21465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:22:11.548902 21465 server_base.cc:1034] running on GCE node
W20260927 11:22:11.548875 21470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:11.548875 21471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:11.548882 21473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:11.549542 21465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:11.549865 21465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:11.551052 21465 hybrid_clock.cc:648] HybridClock initialized: now 1790508131551032 us; error 40 us; skew 500 ppm
Sep 27 11:22:11 dist-test-slave-lsvg krb5kdc[21452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:11.553730 21465 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:22:11.554251 21465 webserver.cc:460] Webserver started at http://127.0.0.1:39259/ using document root <none> and password file <none>
I20260927 11:22:11.554488 21465 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:11.554559 21465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:11.554719 21465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:11.555820 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "178170a7ba23462281f9b58459bc37cc"
format_stamp: "Formatted at 2026-09-27 11:22:11 on dist-test-slave-lsvg"
server_key: "effd5fdeba0a0d6415a44be53e10950c"
server_key_iv: "1e24e624e4b8ea191a34fd28e3440dc0"
server_key_version: "encryptionkey@0"
I20260927 11:22:11.556205 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "178170a7ba23462281f9b58459bc37cc"
format_stamp: "Formatted at 2026-09-27 11:22:11 on dist-test-slave-lsvg"
server_key: "effd5fdeba0a0d6415a44be53e10950c"
server_key_iv: "1e24e624e4b8ea191a34fd28e3440dc0"
server_key_version: "encryptionkey@0"
I20260927 11:22:11.557693 21465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:22:11.558517 21480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:11.558707 21465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:22:11.558812 21465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "178170a7ba23462281f9b58459bc37cc"
format_stamp: "Formatted at 2026-09-27 11:22:11 on dist-test-slave-lsvg"
server_key: "effd5fdeba0a0d6415a44be53e10950c"
server_key_iv: "1e24e624e4b8ea191a34fd28e3440dc0"
server_key_version: "encryptionkey@0"
I20260927 11:22:11.558888 21465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:11.562350 21465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:11.564659 21465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:11.564818 21465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:11.568838 21465 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44775
I20260927 11:22:11.568868 21532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44775 every 8 connection(s)
I20260927 11:22:11.569279 21465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:11.570287 21533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:11.572700 21533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc: Bootstrap starting.
I20260927 11:22:11.573019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 21465
I20260927 11:22:11.573194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:11.573403 392 external_mini_cluster.cc:1428] Setting key c5d775f49020274e3f8e61cf143abf26
I20260927 11:22:11.573547 21533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:11.573909 21533 log.cc:824] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:11.574790 21533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc: No bootstrap required, opened a new log
I20260927 11:22:11.576373 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:11.574002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:37619 (local address 127.0.0.1:44775)
0927 11:22:11.574229 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:11.574245 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:11.574276 (+ 31us) server_negotiation.cc:407] Connection header received
0927 11:22:11.574922 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:11.574933 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:11.574987 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:11.575153 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:11.575889 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:11.575893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:11.575894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:11.575898 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:11.575947 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:11.575961 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:11.576069 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:11.576103 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 11:22:11.576150 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":117,"threads_started":1}
I20260927 11:22:11.577589 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:11.577662 21533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178170a7ba23462281f9b58459bc37cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44775 } }
I20260927 11:22:11.577839 21533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:11.577873 21533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178170a7ba23462281f9b58459bc37cc, State: Initialized, Role: FOLLOWER
I20260927 11:22:11.577992 21533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178170a7ba23462281f9b58459bc37cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44775 } }
I20260927 11:22:11.578114 21533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:11.578152 21533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:11.578208 21533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:11.579023 21533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178170a7ba23462281f9b58459bc37cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44775 } }
I20260927 11:22:11.579141 21533 leader_election.cc:302] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 178170a7ba23462281f9b58459bc37cc; no voters:
I20260927 11:22:11.579355 21533 leader_election.cc:288] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:11.579442 21542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:11.579648 21533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:11.579654 21542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [term 1 LEADER]: Becoming Leader. State: Replica: 178170a7ba23462281f9b58459bc37cc, State: Running, Role: LEADER
I20260927 11:22:11.579842 21542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178170a7ba23462281f9b58459bc37cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44775 } }
I20260927 11:22:11.580129 21542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 178170a7ba23462281f9b58459bc37cc. Latest consensus state: current_term: 1 leader_uuid: "178170a7ba23462281f9b58459bc37cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178170a7ba23462281f9b58459bc37cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44775 } } }
I20260927 11:22:11.580192 21542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:11.580448 21545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "178170a7ba23462281f9b58459bc37cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178170a7ba23462281f9b58459bc37cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44775 } } }
I20260927 11:22:11.580572 21545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:11.580799 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:11.579459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:53891 (local address 127.0.0.1:44775)
0927 11:22:11.579502 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:11.579504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:11.579581 (+ 77us) server_negotiation.cc:407] Connection header received
0927 11:22:11.579652 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:11.579653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:11.579658 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:11.579748 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:11.580562 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:11.580563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:11.580563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:11.580568 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:11.580583 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:11.580588 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:11.580640 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:11.580723 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 11:22:11.580737 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 11:22:11.581913 21559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:22:11.581959 21559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:22:11.582022 21549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:11.582212 21549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:11.583751 21549 catalog_manager.cc:1347] Generated new cluster ID: 164b669fb544493287b180796d0bb6d8
I20260927 11:22:11.583813 21549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:11.596313 21549 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:11.596518 21549 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:11.602593 21549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 178170a7ba23462281f9b58459bc37cc: Generated new TSK 0
I20260927 11:22:11.602816 21549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:11.607503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 21465
2026-09-27T11:22:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:11 dist-test-slave-lsvg krb5kdc[21564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:11 dist-test-slave-lsvg krb5kdc[21564](info): set up 2 sockets
Sep 27 11:22:11 dist-test-slave-lsvg krb5kdc[21564](info): commencing operation
krb5kdc: starting...
W20260927 11:22:12.641340 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:12 dist-test-slave-lsvg krb5kdc[21564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:22:12.760753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:12.843184 21577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:12.843390 21577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:12.843412 21577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:12.844568 21577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:12.844610 21577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:12.844625 21577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:12.844683 21577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:12.844702 21577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:12.846174 21577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46131
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:12.846431 21577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:12.846657 21577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:12.851534 21585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:12.851557 21583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:12.851688 21582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:12.851953 21577 server_base.cc:1034] running on GCE node
I20260927 11:22:12.852218 21577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:12.852579 21577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:12.853745 21577 hybrid_clock.cc:648] HybridClock initialized: now 1790508132853747 us; error 45 us; skew 500 ppm
Sep 27 11:22:12 dist-test-slave-lsvg krb5kdc[21564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:12.860198 21577 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:22:12.860687 21577 webserver.cc:460] Webserver started at http://127.0.0.1:45581/ using document root <none> and password file <none>
I20260927 11:22:12.860983 21577 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:12.861091 21577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:12.861230 21577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:12.862339 21577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "5e157816d395445ba2ac381c0bf6694f"
format_stamp: "Formatted at 2026-09-27 11:22:12 on dist-test-slave-lsvg"
server_key: "8356a7d7859d9dab596b723a7f7c5091"
server_key_iv: "73cf5a16bb1c055589e7fb3e75b922b7"
server_key_version: "encryptionkey@0"
I20260927 11:22:12.862716 21577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e157816d395445ba2ac381c0bf6694f"
format_stamp: "Formatted at 2026-09-27 11:22:12 on dist-test-slave-lsvg"
server_key: "8356a7d7859d9dab596b723a7f7c5091"
server_key_iv: "73cf5a16bb1c055589e7fb3e75b922b7"
server_key_version: "encryptionkey@0"
I20260927 11:22:12.864111 21577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:22:12.864877 21592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:12.865094 21577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:22:12.865185 21577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "5e157816d395445ba2ac381c0bf6694f"
format_stamp: "Formatted at 2026-09-27 11:22:12 on dist-test-slave-lsvg"
server_key: "8356a7d7859d9dab596b723a7f7c5091"
server_key_iv: "73cf5a16bb1c055589e7fb3e75b922b7"
server_key_version: "encryptionkey@0"
I20260927 11:22:12.865276 21577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:12.875867 21577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:12.877890 21577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:12.878054 21577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:12.881901 21577 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46131
I20260927 11:22:12.881932 21644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46131 every 8 connection(s)
I20260927 11:22:12.882299 21577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:12.883208 21645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:12.885427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 21577
I20260927 11:22:12.885576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:12.885715 21645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f: Bootstrap starting.
I20260927 11:22:12.885749 392 external_mini_cluster.cc:1428] Setting key a97c8dfdafb7b7817341581055567abb
I20260927 11:22:12.886462 21645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:12.886790 21645 log.cc:824] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:12.887619 21645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f: No bootstrap required, opened a new log
I20260927 11:22:12.889572 21645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e157816d395445ba2ac381c0bf6694f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } }
I20260927 11:22:12.889776 21645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:12.889817 21645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e157816d395445ba2ac381c0bf6694f, State: Initialized, Role: FOLLOWER
I20260927 11:22:12.889936 21645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e157816d395445ba2ac381c0bf6694f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } }
I20260927 11:22:12.890008 21645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:12.890034 21645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:12.890074 21645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:12.890793 21645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e157816d395445ba2ac381c0bf6694f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } }
I20260927 11:22:12.890884 21645 leader_election.cc:302] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e157816d395445ba2ac381c0bf6694f; no voters:
I20260927 11:22:12.891080 21645 leader_election.cc:288] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:12.891103 21650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:12.891212 21650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [term 1 LEADER]: Becoming Leader. State: Replica: 5e157816d395445ba2ac381c0bf6694f, State: Running, Role: LEADER
I20260927 11:22:12.891304 21650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e157816d395445ba2ac381c0bf6694f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } }
I20260927 11:22:12.891345 21645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:12.891630 21650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e157816d395445ba2ac381c0bf6694f. Latest consensus state: current_term: 1 leader_uuid: "5e157816d395445ba2ac381c0bf6694f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e157816d395445ba2ac381c0bf6694f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } } }
I20260927 11:22:12.891641 21651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e157816d395445ba2ac381c0bf6694f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e157816d395445ba2ac381c0bf6694f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } } }
I20260927 11:22:12.891710 21650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:12.891739 21651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:12.892153 21655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 11:22:12 dist-test-slave-lsvg krb5kdc[21564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 11:22:12.894635 21655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:12.896281 21655 catalog_manager.cc:1347] Generated new cluster ID: 739cb734db29462884fb762f83a97543
I20260927 11:22:12.896337 21655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:12.898516 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:12.886306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:48937 (local address 127.0.0.1:46131)
0927 11:22:12.886504 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:12.886514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:12.886548 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:22:12.887165 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:12.887173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:12.887227 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:12.887398 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:12.888055 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:12.890119 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:12.891159 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:12.891995 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:12.892438 (+ 443us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:12.894744 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:12.894748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:12.894748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:12.894760 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:12.896943 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:12.897377 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:12.897380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:12.897381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:12.897478 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:12.897715 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:12.897717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:12.897717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:12.897947 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:12.898151 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:12.898245 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 11:22:12.898347 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260927 11:22:12.899451 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:12.902707 21655 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:12.903379 21655 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:12.907404 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:12.900760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:54129 (local address 127.0.0.1:46131)
0927 11:22:12.900806 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:12.900808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:12.900917 (+ 109us) server_negotiation.cc:407] Connection header received
0927 11:22:12.900987 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:12.900988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:12.900997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:12.901041 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:12.901544 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:12.902018 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:12.903119 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:12.903354 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:12.903427 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:12.904566 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:12.904567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:12.904568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:12.904574 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:12.905996 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:12.906585 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:12.906586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:12.906587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:12.906664 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:12.906893 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:12.906894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:12.906894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:12.907004 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:12.907119 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:12.907241 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 11:22:12.907302 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 11:22:12.909544 21655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e157816d395445ba2ac381c0bf6694f: Generated new TSK 0
I20260927 11:22:12.909711 21655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:12.911820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 21577
W20260927 11:22:12.916131 21562 connection.cc:535] client connection to 127.0.0.1:46131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46131): Connection reset by peer (error 104)
2026-09-27T11:22:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:12 dist-test-slave-lsvg krb5kdc[21676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:12 dist-test-slave-lsvg krb5kdc[21676](info): set up 2 sockets
Sep 27 11:22:12 dist-test-slave-lsvg krb5kdc[21676](info): commencing operation
krb5kdc: starting...
W20260927 11:22:13.945217 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:14 dist-test-slave-lsvg krb5kdc[21676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:22:14.064256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:14.145215 21689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:14.145440 21689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:14.145465 21689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:14.146639 21689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:14.146680 21689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:14.146695 21689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:14.146708 21689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:14.146720 21689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:14.148161 21689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:14.148447 21689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:14.148712 21689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:14.153697 21695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:14.153719 21689 server_base.cc:1034] running on GCE node
W20260927 11:22:14.153698 21697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:14.153775 21694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:14.154258 21689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:14.154549 21689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:14.155694 21689 hybrid_clock.cc:648] HybridClock initialized: now 1790508134155681 us; error 29 us; skew 500 ppm
Sep 27 11:22:14 dist-test-slave-lsvg krb5kdc[21676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:14.158227 21689 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:22:14.158733 21689 webserver.cc:460] Webserver started at http://127.0.0.1:42539/ using document root <none> and password file <none>
I20260927 11:22:14.158977 21689 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:14.159047 21689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:14.159205 21689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:14.160305 21689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "cbeb10e3f2df4150b1c5ae398a42cf62"
format_stamp: "Formatted at 2026-09-27 11:22:14 on dist-test-slave-lsvg"
server_key: "a979ea8772bd533692d416cfd24205a9"
server_key_iv: "4ecd34ef9b57e7351f983b8e621b10a9"
server_key_version: "encryptionkey@0"
I20260927 11:22:14.160679 21689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "cbeb10e3f2df4150b1c5ae398a42cf62"
format_stamp: "Formatted at 2026-09-27 11:22:14 on dist-test-slave-lsvg"
server_key: "a979ea8772bd533692d416cfd24205a9"
server_key_iv: "4ecd34ef9b57e7351f983b8e621b10a9"
server_key_version: "encryptionkey@0"
I20260927 11:22:14.162093 21689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:22:14.162810 21704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:14.162979 21689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:22:14.163084 21689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "cbeb10e3f2df4150b1c5ae398a42cf62"
format_stamp: "Formatted at 2026-09-27 11:22:14 on dist-test-slave-lsvg"
server_key: "a979ea8772bd533692d416cfd24205a9"
server_key_iv: "4ecd34ef9b57e7351f983b8e621b10a9"
server_key_version: "encryptionkey@0"
I20260927 11:22:14.163167 21689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:14.172708 21689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:14.174633 21689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:14.174795 21689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:14.178824 21689 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41783
I20260927 11:22:14.178855 21756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41783 every 8 connection(s)
I20260927 11:22:14.179199 21689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:14.180128 21757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:14.182281 21757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62: Bootstrap starting.
I20260927 11:22:14.182780 21757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:14.183005 21757 log.cc:824] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:14.183617 21757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62: No bootstrap required, opened a new log
I20260927 11:22:14.185190 21757 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbeb10e3f2df4150b1c5ae398a42cf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41783 } }
I20260927 11:22:14.185321 21757 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:14.185343 21757 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbeb10e3f2df4150b1c5ae398a42cf62, State: Initialized, Role: FOLLOWER
I20260927 11:22:14.185439 21757 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbeb10e3f2df4150b1c5ae398a42cf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41783 } }
I20260927 11:22:14.185490 21757 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:14.185510 21757 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:14.185535 21757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:14.185999 21757 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbeb10e3f2df4150b1c5ae398a42cf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41783 } }
I20260927 11:22:14.186087 21757 leader_election.cc:302] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbeb10e3f2df4150b1c5ae398a42cf62; no voters:
I20260927 11:22:14.186203 21757 leader_election.cc:288] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:14.186252 21760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:14.186414 21760 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [term 1 LEADER]: Becoming Leader. State: Replica: cbeb10e3f2df4150b1c5ae398a42cf62, State: Running, Role: LEADER
I20260927 11:22:14.186440 21757 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:14.186515 21760 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbeb10e3f2df4150b1c5ae398a42cf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41783 } }
I20260927 11:22:14.187062 21762 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbeb10e3f2df4150b1c5ae398a42cf62. Latest consensus state: current_term: 1 leader_uuid: "cbeb10e3f2df4150b1c5ae398a42cf62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbeb10e3f2df4150b1c5ae398a42cf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41783 } } }
I20260927 11:22:14.187163 21762 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:14.187461 21769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:14.187573 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbeb10e3f2df4150b1c5ae398a42cf62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbeb10e3f2df4150b1c5ae398a42cf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41783 } } }
I20260927 11:22:14.187668 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:14.187971 21769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:14.188982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 21689
I20260927 11:22:14.189152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:14.189337 392 external_mini_cluster.cc:1428] Setting key 8353c0ad5897791cb8fe3ce5f8682f83
I20260927 11:22:14.189572 21769 catalog_manager.cc:1347] Generated new cluster ID: efa566acf00a426fb0583b1cc7a7245b
I20260927 11:22:14.189630 21769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:14.195650 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:14.190000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:47963 (local address 127.0.0.1:41783)
0927 11:22:14.190167 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:14.190175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:14.190206 (+ 31us) server_negotiation.cc:407] Connection header received
0927 11:22:14.190319 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:14.190325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:14.190386 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:14.190774 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:14.190868 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:14.192297 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:14.193498 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:14.194112 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:14.194211 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:14.194968 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:14.194973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:14.194973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:14.195179 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:14.195215 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:14.195229 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:14.195304 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:14.195425 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 11:22:14.195452 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260927 11:22:14.196568 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:14.204998 21780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:14.201349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:49055 (local address 127.0.0.1:41783)
0927 11:22:14.201390 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:14.201392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:14.201566 (+ 174us) server_negotiation.cc:407] Connection header received
0927 11:22:14.201602 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:14.201603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:14.201615 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:14.201676 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:14.201814 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:14.202592 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:14.203559 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:14.203902 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:14.203977 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:14.204729 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:14.204731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:14.204731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:14.204736 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:14.204751 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:14.204757 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:14.204814 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:14.204895 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 11:22:14.204909 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":21}
I20260927 11:22:14.228829 21769 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:14.229454 21769 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:14.231997 21769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbeb10e3f2df4150b1c5ae398a42cf62: Generated new TSK 0
I20260927 11:22:14.232213 21769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:14.235386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 21689
W20260927 11:22:14.240199 21674 connection.cc:535] client connection to 127.0.0.1:41783 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41783): Connection reset by peer (error 104)
2026-09-27T11:22:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:14 dist-test-slave-lsvg krb5kdc[21788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:14 dist-test-slave-lsvg krb5kdc[21788](info): set up 2 sockets
Sep 27 11:22:14 dist-test-slave-lsvg krb5kdc[21788](info): commencing operation
krb5kdc: starting...
W20260927 11:22:15.329290 392 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:15 dist-test-slave-lsvg krb5kdc[21788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:22:15.451993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:15.535764 21801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:15.536036 21801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:15.536078 21801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:15.537521 21801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:15.537598 21801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:15.537628 21801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:15.537665 21801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:15.537691 21801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:15.540071 21801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35229
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:15.540392 21801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:15.540660 21801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:15.546336 21806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:15.546358 21809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:15.546336 21807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:15.547580 21801 server_base.cc:1034] running on GCE node
I20260927 11:22:15.547878 21801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:15.548153 21801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:15.549302 21801 hybrid_clock.cc:648] HybridClock initialized: now 1790508135549288 us; error 35 us; skew 500 ppm
Sep 27 11:22:15 dist-test-slave-lsvg krb5kdc[21788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:15.551980 21801 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:22:15.552389 21801 webserver.cc:460] Webserver started at http://127.0.0.1:42363/ using document root <none> and password file <none>
I20260927 11:22:15.552634 21801 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:15.552686 21801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:15.552865 21801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:15.554090 21801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "b9003b9958b14200935eaf16a2e8cd3a"
format_stamp: "Formatted at 2026-09-27 11:22:15 on dist-test-slave-lsvg"
server_key: "dee0db0049bf47bd1697495c762a94d0"
server_key_iv: "d9ea3b13bd2919ac7a44d0692daf6b9b"
server_key_version: "encryptionkey@0"
I20260927 11:22:15.554644 21801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "b9003b9958b14200935eaf16a2e8cd3a"
format_stamp: "Formatted at 2026-09-27 11:22:15 on dist-test-slave-lsvg"
server_key: "dee0db0049bf47bd1697495c762a94d0"
server_key_iv: "d9ea3b13bd2919ac7a44d0692daf6b9b"
server_key_version: "encryptionkey@0"
I20260927 11:22:15.556216 21801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:22:15.557031 21816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:15.557291 21801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:22:15.557430 21801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "b9003b9958b14200935eaf16a2e8cd3a"
format_stamp: "Formatted at 2026-09-27 11:22:15 on dist-test-slave-lsvg"
server_key: "dee0db0049bf47bd1697495c762a94d0"
server_key_iv: "d9ea3b13bd2919ac7a44d0692daf6b9b"
server_key_version: "encryptionkey@0"
I20260927 11:22:15.557526 21801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:15.561630 21801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:15.563791 21801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:15.563966 21801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:15.568094 21801 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35229
I20260927 11:22:15.568182 21868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35229 every 8 connection(s)
I20260927 11:22:15.568512 21801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:15.569550 21869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:15.571779 21869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a: Bootstrap starting.
I20260927 11:22:15.572355 21869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:15.572638 21869 log.cc:824] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:15.573359 21869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a: No bootstrap required, opened a new log
I20260927 11:22:15.574937 21869 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9003b9958b14200935eaf16a2e8cd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35229 } }
I20260927 11:22:15.575071 21869 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:15.575094 21869 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9003b9958b14200935eaf16a2e8cd3a, State: Initialized, Role: FOLLOWER
I20260927 11:22:15.575253 21869 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9003b9958b14200935eaf16a2e8cd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35229 } }
I20260927 11:22:15.575315 21869 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:15.575336 21869 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:15.575403 21869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:15.575911 21869 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9003b9958b14200935eaf16a2e8cd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35229 } }
I20260927 11:22:15.576016 21869 leader_election.cc:302] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9003b9958b14200935eaf16a2e8cd3a; no voters:
I20260927 11:22:15.576195 21869 leader_election.cc:288] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:15.576242 21872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:15.576417 21872 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [term 1 LEADER]: Becoming Leader. State: Replica: b9003b9958b14200935eaf16a2e8cd3a, State: Running, Role: LEADER
I20260927 11:22:15.576450 21869 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:15.576504 21872 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9003b9958b14200935eaf16a2e8cd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35229 } }
I20260927 11:22:15.576675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 21801
I20260927 11:22:15.576790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:15.576978 392 external_mini_cluster.cc:1428] Setting key f4caf12a63956d973cbd63765c00befa
I20260927 11:22:15.577039 21873 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9003b9958b14200935eaf16a2e8cd3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9003b9958b14200935eaf16a2e8cd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35229 } } }
I20260927 11:22:15.577163 21873 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:15.577406 21881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:15.577950 21881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:15.578174 21874 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9003b9958b14200935eaf16a2e8cd3a. Latest consensus state: current_term: 1 leader_uuid: "b9003b9958b14200935eaf16a2e8cd3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9003b9958b14200935eaf16a2e8cd3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35229 } } }
I20260927 11:22:15.578370 21874 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:15.579949 21888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:15.577656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:54723 (local address 127.0.0.1:35229)
0927 11:22:15.577865 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:15.577874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:15.578253 (+ 379us) server_negotiation.cc:407] Connection header received
0927 11:22:15.578472 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:15.578480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:15.578539 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:15.578682 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:15.579463 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:15.579466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:15.579466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:15.579471 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:15.579527 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:15.579541 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:15.579634 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:15.579719 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 11:22:15.579765 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260927 11:22:15.580108 21881 catalog_manager.cc:1347] Generated new cluster ID: 9274d8671e6841cdbb4cadecd2c3f1c7
I20260927 11:22:15.580163 21881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:15.580864 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:15.583446 21888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:15.582300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:36747 (local address 127.0.0.1:35229)
0927 11:22:15.582336 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:15.582338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:15.582520 (+ 182us) server_negotiation.cc:407] Connection header received
0927 11:22:15.582578 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:15.582579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:15.582583 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:15.582650 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:15.583231 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:15.583232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:15.583232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:15.583236 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:15.583251 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:15.583256 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:15.583316 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:15.583367 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 11:22:15.583380 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:22:15.594945 21881 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:15.595113 21881 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:15.598632 21881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9003b9958b14200935eaf16a2e8cd3a: Generated new TSK 0
I20260927 11:22:15.598793 21881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:15.606235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 21801
2026-09-27T11:22:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:15 dist-test-slave-lsvg krb5kdc[21900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:15 dist-test-slave-lsvg krb5kdc[21900](info): set up 2 sockets
Sep 27 11:22:15 dist-test-slave-lsvg krb5kdc[21900](info): commencing operation
krb5kdc: starting...
W20260927 11:22:16.637384 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:16 dist-test-slave-lsvg krb5kdc[21900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:22:16.760578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33823
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:16.858582 21913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:16.858794 21913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:16.858819 21913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:16.859946 21913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:16.859985 21913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:16.860003 21913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:16.860016 21913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:16.860028 21913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:16.861500 21913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:16.861707 21913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:16.861886 21913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:16.866974 21918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:16.867092 21913 server_base.cc:1034] running on GCE node
W20260927 11:22:16.866927 21921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:16.867169 21919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:16.867410 21913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:16.867743 21913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:16.868903 21913 hybrid_clock.cc:648] HybridClock initialized: now 1790508136868895 us; error 31 us; skew 500 ppm
Sep 27 11:22:16 dist-test-slave-lsvg krb5kdc[21900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:16.871426 21913 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:22:16.871899 21913 webserver.cc:460] Webserver started at http://127.0.98.62:44169/ using document root <none> and password file <none>
I20260927 11:22:16.872152 21913 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:16.872221 21913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:16.872371 21913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:16.873502 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "8f8396cca904470d8c06c85c7c1bf700"
format_stamp: "Formatted at 2026-09-27 11:22:16 on dist-test-slave-lsvg"
server_key: "a6b9142e3de838bc4138f299fae60998"
server_key_iv: "9a87cfde3e962f8789148b75b76051e0"
server_key_version: "encryptionkey@0"
I20260927 11:22:16.873849 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f8396cca904470d8c06c85c7c1bf700"
format_stamp: "Formatted at 2026-09-27 11:22:16 on dist-test-slave-lsvg"
server_key: "a6b9142e3de838bc4138f299fae60998"
server_key_iv: "9a87cfde3e962f8789148b75b76051e0"
server_key_version: "encryptionkey@0"
I20260927 11:22:16.875226 21913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:22:16.876041 21928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:16.876219 21913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:22:16.876324 21913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "8f8396cca904470d8c06c85c7c1bf700"
format_stamp: "Formatted at 2026-09-27 11:22:16 on dist-test-slave-lsvg"
server_key: "a6b9142e3de838bc4138f299fae60998"
server_key_iv: "9a87cfde3e962f8789148b75b76051e0"
server_key_version: "encryptionkey@0"
I20260927 11:22:16.876398 21913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:16.903051 21913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:16.905306 21913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:16.905478 21913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:16.909372 21913 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33823
I20260927 11:22:16.909418 21980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33823 every 8 connection(s)
I20260927 11:22:16.909799 21913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:16.910658 21981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:16.912952 21981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700: Bootstrap starting.
I20260927 11:22:16.913553 21981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:16.913832 21981 log.cc:824] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:16.914541 21981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700: No bootstrap required, opened a new log
I20260927 11:22:16.915804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 21913
I20260927 11:22:16.915921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:16.916121 392 external_mini_cluster.cc:1428] Setting key 8c933e0417c212966b12d8b3d0cc23b2
I20260927 11:22:16.916471 21981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8396cca904470d8c06c85c7c1bf700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33823 } }
I20260927 11:22:16.916671 21981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:16.916723 21981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8396cca904470d8c06c85c7c1bf700, State: Initialized, Role: FOLLOWER
I20260927 11:22:16.916869 21981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8396cca904470d8c06c85c7c1bf700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33823 } }
I20260927 11:22:16.916954 21981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:16.916989 21981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:16.917038 21981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:16.918259 21981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8396cca904470d8c06c85c7c1bf700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33823 } }
I20260927 11:22:16.918380 21981 leader_election.cc:302] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f8396cca904470d8c06c85c7c1bf700; no voters:
I20260927 11:22:16.918534 21981 leader_election.cc:288] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:16.918596 21986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:16.918725 21986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [term 1 LEADER]: Becoming Leader. State: Replica: 8f8396cca904470d8c06c85c7c1bf700, State: Running, Role: LEADER
I20260927 11:22:16.918795 21981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:16.918809 21986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8396cca904470d8c06c85c7c1bf700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33823 } }
I20260927 11:22:16.919220 21986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f8396cca904470d8c06c85c7c1bf700. Latest consensus state: current_term: 1 leader_uuid: "8f8396cca904470d8c06c85c7c1bf700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8396cca904470d8c06c85c7c1bf700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33823 } } }
I20260927 11:22:16.919281 21986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:16.919438 21987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f8396cca904470d8c06c85c7c1bf700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8396cca904470d8c06c85c7c1bf700" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33823 } } }
I20260927 11:22:16.919554 21987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700 [sys.catalog]: This master's current role is: LEADER
W20260927 11:22:16.920644 22002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:22:16.920704 22002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:22:16.920745 21993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:16.920884 21993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:16.922468 21993 catalog_manager.cc:1347] Generated new cluster ID: 66f43f42493c449ab1619d1bc3f5ce15
I20260927 11:22:16.922533 21993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:22:16 dist-test-slave-lsvg krb5kdc[21900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:22:16.929586 21993 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:16.930279 21993 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:16.930524 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:16.916650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39105 (local address 127.0.98.62:33823)
0927 11:22:16.916878 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:16.916887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:16.916927 (+ 40us) server_negotiation.cc:407] Connection header received
0927 11:22:16.917090 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:16.917097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:16.917156 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:16.917279 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:16.918159 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:16.919417 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:16.920549 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:16.921126 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:16.921257 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:16.926066 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:16.926070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:16.926070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:16.926082 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:16.928615 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:16.929105 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:16.929108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:16.929109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:16.929216 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:16.929499 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:16.929500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:16.929500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:16.929801 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:16.929977 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:16.930107 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 11:22:16.930262 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":176,"thread_start_us":84,"threads_started":1}
I20260927 11:22:16.931416 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:16.939430 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:16.932750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42723 (local address 127.0.98.62:33823)
0927 11:22:16.932785 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:16.932787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:16.933293 (+ 506us) server_negotiation.cc:407] Connection header received
0927 11:22:16.933379 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:16.933379 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:16.933390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:16.933447 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:16.933987 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:16.934520 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:16.935265 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:16.935487 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:16.935558 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:16.936753 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:16.936754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:16.936754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:16.936763 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:16.938271 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:16.938643 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:16.938644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:16.938644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:16.938703 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:16.938934 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:16.938935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:16.938935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:16.939038 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:16.939144 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:16.939264 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 11:22:16.939325 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:22:16.943020 21993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f8396cca904470d8c06c85c7c1bf700: Generated new TSK 0
I20260927 11:22:16.943210 21993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:16.947329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 21913
W20260927 11:22:16.951444 21898 connection.cc:535] client connection to 127.0.98.62:33823 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33823): Connection reset by peer (error 104)
2026-09-27T11:22:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:17 dist-test-slave-lsvg krb5kdc[22012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:17 dist-test-slave-lsvg krb5kdc[22012](info): set up 2 sockets
Sep 27 11:22:17 dist-test-slave-lsvg krb5kdc[22012](info): commencing operation
krb5kdc: starting...
W20260927 11:22:17.985493 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:18 dist-test-slave-lsvg krb5kdc[22012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:22:18.108487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42465
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:18.189961 22025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:18.190196 22025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:18.190232 22025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:18.191347 22025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:18.191416 22025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:18.191442 22025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:18.191481 22025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:18.191505 22025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:18.192932 22025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:18.193217 22025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:18.193439 22025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:18.198218 22031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:18.198174 22030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:18.198184 22033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:18.198560 22025 server_base.cc:1034] running on GCE node
I20260927 11:22:18.198793 22025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:18.199070 22025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:18.200210 22025 hybrid_clock.cc:648] HybridClock initialized: now 1790508138200189 us; error 37 us; skew 500 ppm
Sep 27 11:22:18 dist-test-slave-lsvg krb5kdc[22012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:18.206283 22025 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:22:18.206761 22025 webserver.cc:460] Webserver started at http://127.0.98.62:38683/ using document root <none> and password file <none>
I20260927 11:22:18.206962 22025 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:18.207006 22025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:18.207093 22025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:18.208115 22025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "3e09ce03d4744de5a7b0d1f840c6488d"
format_stamp: "Formatted at 2026-09-27 11:22:18 on dist-test-slave-lsvg"
server_key: "7959bb662161710f6978d33f339addcd"
server_key_iv: "28fe227dfecb6db436e6acc0aca826a8"
server_key_version: "encryptionkey@0"
I20260927 11:22:18.208413 22025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "3e09ce03d4744de5a7b0d1f840c6488d"
format_stamp: "Formatted at 2026-09-27 11:22:18 on dist-test-slave-lsvg"
server_key: "7959bb662161710f6978d33f339addcd"
server_key_iv: "28fe227dfecb6db436e6acc0aca826a8"
server_key_version: "encryptionkey@0"
I20260927 11:22:18.209759 22025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:22:18.210436 22040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:18.210598 22025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:22:18.210702 22025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "3e09ce03d4744de5a7b0d1f840c6488d"
format_stamp: "Formatted at 2026-09-27 11:22:18 on dist-test-slave-lsvg"
server_key: "7959bb662161710f6978d33f339addcd"
server_key_iv: "28fe227dfecb6db436e6acc0aca826a8"
server_key_version: "encryptionkey@0"
I20260927 11:22:18.210779 22025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:18.226332 22025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:18.228763 22025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:18.228935 22025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:18.233032 22025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42465
I20260927 11:22:18.233122 22092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42465 every 8 connection(s)
I20260927 11:22:18.233497 22025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:18.234431 22093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:18.236860 22093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d: Bootstrap starting.
I20260927 11:22:18.237531 22093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:18.237778 22093 log.cc:824] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:18.238502 22093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d: No bootstrap required, opened a new log
I20260927 11:22:18.240161 22093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e09ce03d4744de5a7b0d1f840c6488d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42465 } }
I20260927 11:22:18.240306 22093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:18.240340 22093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e09ce03d4744de5a7b0d1f840c6488d, State: Initialized, Role: FOLLOWER
I20260927 11:22:18.240454 22093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e09ce03d4744de5a7b0d1f840c6488d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42465 } }
I20260927 11:22:18.240511 22093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:18.240533 22093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:18.240615 22093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:18.241211 22093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e09ce03d4744de5a7b0d1f840c6488d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42465 } }
I20260927 11:22:18.241343 22093 leader_election.cc:302] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e09ce03d4744de5a7b0d1f840c6488d; no voters:
I20260927 11:22:18.241539 22093 leader_election.cc:288] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:18.241573 22096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:18.241750 22096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [term 1 LEADER]: Becoming Leader. State: Replica: 3e09ce03d4744de5a7b0d1f840c6488d, State: Running, Role: LEADER
I20260927 11:22:18.241816 22093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:18.241884 22096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e09ce03d4744de5a7b0d1f840c6488d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42465 } }
I20260927 11:22:18.242249 22098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e09ce03d4744de5a7b0d1f840c6488d. Latest consensus state: current_term: 1 leader_uuid: "3e09ce03d4744de5a7b0d1f840c6488d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e09ce03d4744de5a7b0d1f840c6488d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42465 } } }
I20260927 11:22:18.242370 22098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:18.242383 22097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e09ce03d4744de5a7b0d1f840c6488d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e09ce03d4744de5a7b0d1f840c6488d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42465 } } }
I20260927 11:22:18.242528 22097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:18.242887 22105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:18.243378 22105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:18.243676 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 22025
I20260927 11:22:18.243798 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:18.244000 392 external_mini_cluster.cc:1428] Setting key 5373914c0b4b5b254352f91519b0f7e7
I20260927 11:22:18.245179 22105 catalog_manager.cc:1347] Generated new cluster ID: e99b984ac6a7494e846e7903a436bc0f
I20260927 11:22:18.245234 22105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:18.250964 22105 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:18.251498 22105 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:18.255066 22105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e09ce03d4744de5a7b0d1f840c6488d: Generated new TSK 0
I20260927 11:22:18.255246 22105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:18.256018 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:18.244452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37797 (local address 127.0.98.62:42465)
0927 11:22:18.249511 (+ 5059us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:18.249521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:18.249564 (+ 43us) server_negotiation.cc:407] Connection header received
0927 11:22:18.249707 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:18.249714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:18.249773 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:18.249948 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:18.250338 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:18.252329 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:18.253524 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:18.254230 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:18.254357 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:18.255279 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:18.255284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:18.255284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:18.255513 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:18.255548 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:18.255561 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:18.255675 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:18.255781 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 11:22:18.255821 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":102,"threads_started":1}
I20260927 11:22:18.257084 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:18.261677 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:18.258356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53031 (local address 127.0.98.62:42465)
0927 11:22:18.258389 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:18.258391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:18.258504 (+ 113us) server_negotiation.cc:407] Connection header received
0927 11:22:18.258567 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:18.258568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:18.258576 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:18.258623 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:18.258816 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:18.259303 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:18.260456 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:18.260653 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:18.260700 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:18.261440 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:18.261442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:18.261442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:18.261446 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:18.261463 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:18.261467 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:18.261515 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:18.261617 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 11:22:18.261628 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 11:22:18.263669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 22025
W20260927 11:22:18.267556 22010 connection.cc:535] client connection to 127.0.98.62:42465 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42465): Connection reset by peer (error 104)
2026-09-27T11:22:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:18 dist-test-slave-lsvg krb5kdc[22124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:18 dist-test-slave-lsvg krb5kdc[22124](info): set up 2 sockets
Sep 27 11:22:18 dist-test-slave-lsvg krb5kdc[22124](info): commencing operation
krb5kdc: starting...
W20260927 11:22:19.309279 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:19 dist-test-slave-lsvg krb5kdc[22124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:22:19.432125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39051
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:19.513145 22137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:19.513393 22137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:19.513432 22137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:19.514566 22137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:19.514636 22137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:19.514662 22137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:19.514691 22137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:19.514714 22137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:19.516153 22137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:19.516485 22137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:19.516710 22137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:19.521593 22145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:19.521622 22142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:19.521723 22137 server_base.cc:1034] running on GCE node
W20260927 11:22:19.521606 22143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:19.522121 22137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:19.522419 22137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:19.523631 22137 hybrid_clock.cc:648] HybridClock initialized: now 1790508139523605 us; error 41 us; skew 500 ppm
Sep 27 11:22:19 dist-test-slave-lsvg krb5kdc[22124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:19.527096 22137 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:22:19.527639 22137 webserver.cc:460] Webserver started at http://127.0.98.62:44485/ using document root <none> and password file <none>
I20260927 11:22:19.527889 22137 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:19.527962 22137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:19.528134 22137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:19.529345 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "3ff38475beec4ec787c42de5810e52b1"
format_stamp: "Formatted at 2026-09-27 11:22:19 on dist-test-slave-lsvg"
server_key: "c2e3812bc186b9f77feffd43a1ac0549"
server_key_iv: "bbd2633d7ab85e66c09bf09d28d77f89"
server_key_version: "encryptionkey@0"
I20260927 11:22:19.529742 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ff38475beec4ec787c42de5810e52b1"
format_stamp: "Formatted at 2026-09-27 11:22:19 on dist-test-slave-lsvg"
server_key: "c2e3812bc186b9f77feffd43a1ac0549"
server_key_iv: "bbd2633d7ab85e66c09bf09d28d77f89"
server_key_version: "encryptionkey@0"
I20260927 11:22:19.531189 22137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:22:19.531985 22152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:19.532223 22137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:22:19.532333 22137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "3ff38475beec4ec787c42de5810e52b1"
format_stamp: "Formatted at 2026-09-27 11:22:19 on dist-test-slave-lsvg"
server_key: "c2e3812bc186b9f77feffd43a1ac0549"
server_key_iv: "bbd2633d7ab85e66c09bf09d28d77f89"
server_key_version: "encryptionkey@0"
I20260927 11:22:19.532418 22137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:19.537968 22137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:19.540762 22137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:19.540910 22137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:19.544890 22137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39051
I20260927 11:22:19.544919 22204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39051 every 8 connection(s)
I20260927 11:22:19.545322 22137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:19.546373 22205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:19.546653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 22137
I20260927 11:22:19.546779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:19.546978 392 external_mini_cluster.cc:1428] Setting key e8c9ab01ebac93dd55c5d7698b862f63
I20260927 11:22:19.549741 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:19.547531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53105 (local address 127.0.98.62:39051)
0927 11:22:19.547746 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:19.547756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:19.547798 (+ 42us) server_negotiation.cc:407] Connection header received
0927 11:22:19.548269 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:19.548279 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:19.548324 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:19.548490 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:19.549174 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:19.549178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:19.549179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:19.549184 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:19.549230 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:19.549244 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:19.549325 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:19.549361 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 11:22:19.549510 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260927 11:22:19.550191 22205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1: Bootstrap starting.
I20260927 11:22:19.550807 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:19.550984 22205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:19.551265 22205 log.cc:824] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:19.551998 22205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1: No bootstrap required, opened a new log
I20260927 11:22:19.553244 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:19.552145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48185 (local address 127.0.98.62:39051)
0927 11:22:19.552178 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:19.552180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:19.552270 (+ 90us) server_negotiation.cc:407] Connection header received
0927 11:22:19.552312 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:19.552313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:19.552317 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:19.552382 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:19.553016 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:19.553017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:19.553017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:19.553023 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:19.553038 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:19.553042 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:19.553129 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:19.553175 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 11:22:19.553188 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:22:19.554312 22205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff38475beec4ec787c42de5810e52b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39051 } }
I20260927 11:22:19.554443 22205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:19.554467 22205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff38475beec4ec787c42de5810e52b1, State: Initialized, Role: FOLLOWER
I20260927 11:22:19.554582 22205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff38475beec4ec787c42de5810e52b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39051 } }
I20260927 11:22:19.554666 22205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:19.554689 22205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:19.554718 22205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:19.555212 22205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff38475beec4ec787c42de5810e52b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39051 } }
I20260927 11:22:19.555294 22205 leader_election.cc:302] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ff38475beec4ec787c42de5810e52b1; no voters:
I20260927 11:22:19.555409 22205 leader_election.cc:288] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:19.555462 22215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:19.555605 22215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [term 1 LEADER]: Becoming Leader. State: Replica: 3ff38475beec4ec787c42de5810e52b1, State: Running, Role: LEADER
I20260927 11:22:19.555631 22205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:19.555727 22215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff38475beec4ec787c42de5810e52b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39051 } }
I20260927 11:22:19.556108 22216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ff38475beec4ec787c42de5810e52b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff38475beec4ec787c42de5810e52b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39051 } } }
I20260927 11:22:19.556118 22217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ff38475beec4ec787c42de5810e52b1. Latest consensus state: current_term: 1 leader_uuid: "3ff38475beec4ec787c42de5810e52b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff38475beec4ec787c42de5810e52b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39051 } } }
I20260927 11:22:19.556255 22216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:19.556293 22217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:19.556499 22224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:19.557003 22224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:19.558426 22224 catalog_manager.cc:1347] Generated new cluster ID: e4f2138fdfc64d52b08bbcea36ce5ed1
I20260927 11:22:19.558502 22224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:19.568979 22224 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:19.569198 22224 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:19.580256 22224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ff38475beec4ec787c42de5810e52b1: Generated new TSK 0
I20260927 11:22:19.580595 22224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:19.589250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 22137
2026-09-27T11:22:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:19 dist-test-slave-lsvg krb5kdc[22236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:19 dist-test-slave-lsvg krb5kdc[22236](info): set up 2 sockets
Sep 27 11:22:19 dist-test-slave-lsvg krb5kdc[22236](info): commencing operation
krb5kdc: starting...
W20260927 11:22:20.625377 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:20 dist-test-slave-lsvg krb5kdc[22236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:22:20.746050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36119
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:20.828647 22249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:20.828876 22249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:20.828899 22249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:20.830116 22249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:20.830159 22249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:20.830174 22249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:20.830186 22249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:20.830245 22249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:20.831776 22249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:20.832042 22249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:20.832274 22249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:20.837132 22257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:20.837121 22255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:20.837121 22254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:20.837386 22249 server_base.cc:1034] running on GCE node
I20260927 11:22:20.837612 22249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:20.837898 22249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:20.839051 22249 hybrid_clock.cc:648] HybridClock initialized: now 1790508140839034 us; error 38 us; skew 500 ppm
Sep 27 11:22:20 dist-test-slave-lsvg krb5kdc[22236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:20.841684 22249 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:22:20.842154 22249 webserver.cc:460] Webserver started at http://127.0.98.62:45451/ using document root <none> and password file <none>
I20260927 11:22:20.842386 22249 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:20.842430 22249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:20.842594 22249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:20.843650 22249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "1dff40a3cc11417fbe4083a193e6ff4a"
format_stamp: "Formatted at 2026-09-27 11:22:20 on dist-test-slave-lsvg"
server_key: "b3b6e3fe748252ab09925ef2e7cd7acd"
server_key_iv: "ec721fef974ed3b740349fbc2e6fb3e5"
server_key_version: "encryptionkey@0"
I20260927 11:22:20.843997 22249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "1dff40a3cc11417fbe4083a193e6ff4a"
format_stamp: "Formatted at 2026-09-27 11:22:20 on dist-test-slave-lsvg"
server_key: "b3b6e3fe748252ab09925ef2e7cd7acd"
server_key_iv: "ec721fef974ed3b740349fbc2e6fb3e5"
server_key_version: "encryptionkey@0"
I20260927 11:22:20.845341 22249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:22:20.846098 22264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:20.846285 22249 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:22:20.846375 22249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "1dff40a3cc11417fbe4083a193e6ff4a"
format_stamp: "Formatted at 2026-09-27 11:22:20 on dist-test-slave-lsvg"
server_key: "b3b6e3fe748252ab09925ef2e7cd7acd"
server_key_iv: "ec721fef974ed3b740349fbc2e6fb3e5"
server_key_version: "encryptionkey@0"
I20260927 11:22:20.846468 22249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:20.870549 22249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:20.872902 22249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:20.873049 22249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:20.877200 22249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36119
I20260927 11:22:20.877261 22316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36119 every 8 connection(s)
I20260927 11:22:20.877607 22249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:20.878499 22317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:20.880844 22317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a: Bootstrap starting.
I20260927 11:22:20.881470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 22249
I20260927 11:22:20.881592 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:20.881625 22317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:20.881812 392 external_mini_cluster.cc:1428] Setting key 999cc9d45ea8788123b874d8cde750e7
I20260927 11:22:20.881950 22317 log.cc:824] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:20.882778 22317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a: No bootstrap required, opened a new log
I20260927 11:22:20.885113 22317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dff40a3cc11417fbe4083a193e6ff4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36119 } }
I20260927 11:22:20.885290 22317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:20.885324 22317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dff40a3cc11417fbe4083a193e6ff4a, State: Initialized, Role: FOLLOWER
I20260927 11:22:20.885452 22317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dff40a3cc11417fbe4083a193e6ff4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36119 } }
I20260927 11:22:20.885533 22317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:20.885572 22317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:20.885617 22317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:20.886260 22317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dff40a3cc11417fbe4083a193e6ff4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36119 } }
I20260927 11:22:20.886350 22317 leader_election.cc:302] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dff40a3cc11417fbe4083a193e6ff4a; no voters:
I20260927 11:22:20.886605 22317 leader_election.cc:288] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:20.886698 22322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:20.886870 22322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [term 1 LEADER]: Becoming Leader. State: Replica: 1dff40a3cc11417fbe4083a193e6ff4a, State: Running, Role: LEADER
I20260927 11:22:20.886884 22317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:20.888361 22322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dff40a3cc11417fbe4083a193e6ff4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36119 } }
Sep 27 11:22:20 dist-test-slave-lsvg krb5kdc[22236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:22:20.889142 22324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dff40a3cc11417fbe4083a193e6ff4a. Latest consensus state: current_term: 1 leader_uuid: "1dff40a3cc11417fbe4083a193e6ff4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dff40a3cc11417fbe4083a193e6ff4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36119 } } }
I20260927 11:22:20.889422 22324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:20.889285 22323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dff40a3cc11417fbe4083a193e6ff4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dff40a3cc11417fbe4083a193e6ff4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36119 } } }
I20260927 11:22:20.889578 22323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a [sys.catalog]: This master's current role is: LEADER
W20260927 11:22:20.890048 22336 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:22:20.890102 22336 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:22:20.890151 22337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:20.890285 22337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:20.892210 22337 catalog_manager.cc:1347] Generated new cluster ID: 60a521132cef46678d84b461a5c1171a
I20260927 11:22:20.892261 22337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:20.893155 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:20.882311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44941 (local address 127.0.98.62:36119)
0927 11:22:20.882495 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:20.882504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:20.882568 (+ 64us) server_negotiation.cc:407] Connection header received
0927 11:22:20.883066 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:20.883073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:20.883123 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:20.883233 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:20.884035 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:20.884903 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:20.886007 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:20.886510 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:20.886626 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:20.889432 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:20.889436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:20.889436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:20.889449 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:20.891445 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:20.891847 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:20.891850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:20.891850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:20.891951 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:20.892181 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:20.892182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:20.892182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:20.892494 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:20.892626 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:20.892801 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 11:22:20.892935 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260927 11:22:20.894107 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:20.904458 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:20.895629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55521 (local address 127.0.98.62:36119)
0927 11:22:20.895670 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:20.895672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:20.897256 (+ 1584us) server_negotiation.cc:407] Connection header received
0927 11:22:20.897305 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:20.897306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:20.897318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:20.897376 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:20.898003 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:20.898774 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:20.899785 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:20.900131 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:20.900208 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:20.901660 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:20.901662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:20.901662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:20.901670 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:20.903319 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:20.903686 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:20.903687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:20.903688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:20.903749 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:20.903937 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:20.903938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:20.903938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:20.904077 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:20.904175 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:20.904275 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 11:22:20.904359 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:22:20.915422 22337 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:20.915977 22337 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:20.920384 22337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dff40a3cc11417fbe4083a193e6ff4a: Generated new TSK 0
I20260927 11:22:20.920599 22337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:20.926316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 22249
2026-09-27T11:22:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:20 dist-test-slave-lsvg krb5kdc[22348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:20 dist-test-slave-lsvg krb5kdc[22348](info): set up 2 sockets
Sep 27 11:22:20 dist-test-slave-lsvg krb5kdc[22348](info): commencing operation
krb5kdc: starting...
W20260927 11:22:21.961334 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:22 dist-test-slave-lsvg krb5kdc[22348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:22:22.088124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40135
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:22.169029 22361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:22.169309 22361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:22.169358 22361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:22.170575 22361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:22.170699 22361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:22.170727 22361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:22.170758 22361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:22.170779 22361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:22.172287 22361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:22.172580 22361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:22.172813 22361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:22.177767 22366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:22.177906 22367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:22.177983 22361 server_base.cc:1034] running on GCE node
W20260927 11:22:22.177793 22369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:22.178294 22361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:22.178548 22361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:22.179692 22361 hybrid_clock.cc:648] HybridClock initialized: now 1790508142179673 us; error 44 us; skew 500 ppm
Sep 27 11:22:22 dist-test-slave-lsvg krb5kdc[22348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:22.182296 22361 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:22:22.182793 22361 webserver.cc:460] Webserver started at http://127.0.98.62:41469/ using document root <none> and password file <none>
I20260927 11:22:22.183043 22361 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:22.183089 22361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:22.183236 22361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:22.184294 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "49359d48b0664189b82663289f122580"
format_stamp: "Formatted at 2026-09-27 11:22:22 on dist-test-slave-lsvg"
server_key: "7c858afebc7c3a3a069fa260c4fedc56"
server_key_iv: "df7fe58670ce0ceb1cf2c7b243fd8e9e"
server_key_version: "encryptionkey@0"
I20260927 11:22:22.184664 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "49359d48b0664189b82663289f122580"
format_stamp: "Formatted at 2026-09-27 11:22:22 on dist-test-slave-lsvg"
server_key: "7c858afebc7c3a3a069fa260c4fedc56"
server_key_iv: "df7fe58670ce0ceb1cf2c7b243fd8e9e"
server_key_version: "encryptionkey@0"
I20260927 11:22:22.186039 22361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:22:22.186769 22376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:22.186936 22361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:22:22.187028 22361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "49359d48b0664189b82663289f122580"
format_stamp: "Formatted at 2026-09-27 11:22:22 on dist-test-slave-lsvg"
server_key: "7c858afebc7c3a3a069fa260c4fedc56"
server_key_iv: "df7fe58670ce0ceb1cf2c7b243fd8e9e"
server_key_version: "encryptionkey@0"
I20260927 11:22:22.187116 22361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:22.228552 22361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:22.231050 22361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:22.231221 22361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:22.235344 22361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40135
I20260927 11:22:22.235380 22428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40135 every 8 connection(s)
I20260927 11:22:22.235760 22361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:22.236626 22429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:22.239007 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580: Bootstrap starting.
I20260927 11:22:22.239553 22429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:22.239785 22429 log.cc:824] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:22.240432 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580: No bootstrap required, opened a new log
I20260927 11:22:22.242055 22429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49359d48b0664189b82663289f122580" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40135 } }
I20260927 11:22:22.242199 22429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:22.242221 22429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49359d48b0664189b82663289f122580, State: Initialized, Role: FOLLOWER
I20260927 11:22:22.242372 22429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49359d48b0664189b82663289f122580" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40135 } }
I20260927 11:22:22.242458 22429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:22.242480 22429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:22.242511 22429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:22.243007 22429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49359d48b0664189b82663289f122580" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40135 } }
I20260927 11:22:22.243096 22429 leader_election.cc:302] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49359d48b0664189b82663289f122580; no voters:
I20260927 11:22:22.243225 22429 leader_election.cc:288] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:22.243323 22432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:22.243505 22432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [term 1 LEADER]: Becoming Leader. State: Replica: 49359d48b0664189b82663289f122580, State: Running, Role: LEADER
I20260927 11:22:22.243503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 22361
I20260927 11:22:22.243533 22429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:22.243608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:22.243810 392 external_mini_cluster.cc:1428] Setting key 56afa0d4965610102cb5884aeed4f67c
I20260927 11:22:22.244369 22432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49359d48b0664189b82663289f122580" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40135 } }
I20260927 11:22:22.245098 22433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49359d48b0664189b82663289f122580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49359d48b0664189b82663289f122580" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40135 } } }
I20260927 11:22:22.245252 22433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:22.245446 22434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49359d48b0664189b82663289f122580. Latest consensus state: current_term: 1 leader_uuid: "49359d48b0664189b82663289f122580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49359d48b0664189b82663289f122580" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40135 } } }
I20260927 11:22:22.245527 22434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:22.245615 22449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:22.246040 22449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:22.247414 22449 catalog_manager.cc:1347] Generated new cluster ID: 8f4ee3a9f7d6403fbffe8b9488a3e25c
I20260927 11:22:22.247476 22449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:22.250483 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:22.244481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39381 (local address 127.0.98.62:40135)
0927 11:22:22.244818 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:22.244829 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:22.244862 (+ 33us) server_negotiation.cc:407] Connection header received
0927 11:22:22.244981 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:22.244989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:22.245108 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:22.245523 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:22.245650 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:22.247048 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:22.248169 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:22.248798 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:22.248916 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:22.249653 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:22.249657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:22.249658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:22.249878 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:22.249917 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:22.249934 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:22.250006 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:22.250102 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 11:22:22.250273 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":77,"threads_started":1}
I20260927 11:22:22.251389 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:22.256474 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:22.252973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46489 (local address 127.0.98.62:40135)
0927 11:22:22.253004 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:22.253006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:22.253067 (+ 61us) server_negotiation.cc:407] Connection header received
0927 11:22:22.253190 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:22.253191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:22.253200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:22.253247 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:22.253393 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:22.254129 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:22.255192 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:22.255473 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:22.255531 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:22.256224 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:22.256225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:22.256225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:22.256230 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:22.256243 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:22.256247 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:22.256300 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:22.256379 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 11:22:22.256403 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 11:22:22.264652 22449 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:22.265334 22449 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:22.273547 22449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49359d48b0664189b82663289f122580: Generated new TSK 0
I20260927 11:22:22.273762 22449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:22.280275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 22361
2026-09-27T11:22:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:22 dist-test-slave-lsvg krb5kdc[22460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:22 dist-test-slave-lsvg krb5kdc[22460](info): set up 2 sockets
Sep 27 11:22:22 dist-test-slave-lsvg krb5kdc[22460](info): commencing operation
krb5kdc: starting...
W20260927 11:22:23.321314 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:23 dist-test-slave-lsvg krb5kdc[22460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:22:23.440644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34357
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:23.520838 22473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:23.521062 22473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:23.521086 22473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:23.522294 22473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:23.522341 22473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:23.522354 22473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:23.522367 22473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:23.522379 22473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:23.523808 22473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:23.524004 22473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:23.524173 22473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:23.528939 22481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:23.528954 22479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:23.528999 22478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:23.529227 22473 server_base.cc:1034] running on GCE node
I20260927 11:22:23.529493 22473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:23.529738 22473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:23.530874 22473 hybrid_clock.cc:648] HybridClock initialized: now 1790508143530867 us; error 32 us; skew 500 ppm
Sep 27 11:22:23 dist-test-slave-lsvg krb5kdc[22460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:23.541972 22473 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:22:23.542559 22473 webserver.cc:460] Webserver started at http://127.0.98.62:37167/ using document root <none> and password file <none>
I20260927 11:22:23.542811 22473 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:23.542858 22473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:23.542959 22473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:23.543984 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "4da57cf61769447ca3f783273605617d"
format_stamp: "Formatted at 2026-09-27 11:22:23 on dist-test-slave-lsvg"
server_key: "639c8231fca1b02926ec8e9880ea9530"
server_key_iv: "8414f31358a371c8fa77dcb97fc65dee"
server_key_version: "encryptionkey@0"
I20260927 11:22:23.544281 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "4da57cf61769447ca3f783273605617d"
format_stamp: "Formatted at 2026-09-27 11:22:23 on dist-test-slave-lsvg"
server_key: "639c8231fca1b02926ec8e9880ea9530"
server_key_iv: "8414f31358a371c8fa77dcb97fc65dee"
server_key_version: "encryptionkey@0"
I20260927 11:22:23.545637 22473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:22:23.546406 22488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:23.546600 22473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:22:23.546700 22473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "4da57cf61769447ca3f783273605617d"
format_stamp: "Formatted at 2026-09-27 11:22:23 on dist-test-slave-lsvg"
server_key: "639c8231fca1b02926ec8e9880ea9530"
server_key_iv: "8414f31358a371c8fa77dcb97fc65dee"
server_key_version: "encryptionkey@0"
I20260927 11:22:23.546780 22473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:23.550181 22473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:23.552405 22473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:23.552592 22473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:23.556349 22473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34357
I20260927 11:22:23.556420 22540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34357 every 8 connection(s)
I20260927 11:22:23.556736 22473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:23.557802 22541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:23.560103 22541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d: Bootstrap starting.
I20260927 11:22:23.560696 22541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:23.560981 22541 log.cc:824] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:23.561718 22541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d: No bootstrap required, opened a new log
I20260927 11:22:23.563383 22541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da57cf61769447ca3f783273605617d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34357 } }
I20260927 11:22:23.563562 22541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:22:23.563614 22541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da57cf61769447ca3f783273605617d, State: Initialized, Role: FOLLOWER
I20260927 11:22:23.563759 22541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da57cf61769447ca3f783273605617d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34357 } }
I20260927 11:22:23.563843 22541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:22:23.563884 22541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:22:23.563935 22541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:23.564472 22541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da57cf61769447ca3f783273605617d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34357 } }
I20260927 11:22:23.564591 22541 leader_election.cc:302] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4da57cf61769447ca3f783273605617d; no voters:
I20260927 11:22:23.564778 22541 leader_election.cc:288] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:22:23.564811 22544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:23.564982 22544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [term 1 LEADER]: Becoming Leader. State: Replica: 4da57cf61769447ca3f783273605617d, State: Running, Role: LEADER
I20260927 11:22:23.565034 22541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:23.565207 22544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da57cf61769447ca3f783273605617d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34357 } }
I20260927 11:22:23.565279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 22473
I20260927 11:22:23.565400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:23.565580 392 external_mini_cluster.cc:1428] Setting key 49b6a81bd68b9a030cc6a4b2aac0bf1a
I20260927 11:22:23.566339 22546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4da57cf61769447ca3f783273605617d. Latest consensus state: current_term: 1 leader_uuid: "4da57cf61769447ca3f783273605617d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da57cf61769447ca3f783273605617d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34357 } } }
I20260927 11:22:23.566404 22546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:23.566589 22557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:23.566643 22545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4da57cf61769447ca3f783273605617d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da57cf61769447ca3f783273605617d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34357 } } }
I20260927 11:22:23.566710 22545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:23.567117 22557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:23.568622 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:23.566191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43391 (local address 127.0.98.62:34357)
0927 11:22:23.566933 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:23.566942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:23.566976 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:22:23.567094 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:23.567102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:23.567133 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:23.567605 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:23.568074 (+ 469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:23.568077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:23.568077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:23.568081 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:23.568119 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:23.568133 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:23.568355 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:23.568388 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 11:22:23.568433 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":88,"threads_started":1}
I20260927 11:22:23.569454 22557 catalog_manager.cc:1347] Generated new cluster ID: 57897e9440e54603b322e394dcf361de
I20260927 11:22:23.569535 22557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:23.569691 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:23.582412 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:23.571155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37531 (local address 127.0.98.62:34357)
0927 11:22:23.571216 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:23.571218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:23.581314 (+ 10096us) server_negotiation.cc:407] Connection header received
0927 11:22:23.581406 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:23.581407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:23.581426 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:23.581493 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:23.582192 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:23.582193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:23.582194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:22:23.582197 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:23.582209 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:22:23.582213 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:23.582257 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:23.582310 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 11:22:23.582324 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:22:23.586915 22557 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:23.587101 22557 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:23.593266 22557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4da57cf61769447ca3f783273605617d: Generated new TSK 0
I20260927 11:22:23.593457 22557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:23.596670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 22473
2026-09-27T11:22:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1316 ms)
[----------] 18 tests from AuthTokenIssuingTest (23961 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:23 dist-test-slave-lsvg krb5kdc[22572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:23 dist-test-slave-lsvg krb5kdc[22572](info): set up 2 sockets
Sep 27 11:22:23 dist-test-slave-lsvg krb5kdc[22572](info): commencing operation
krb5kdc: starting...
W20260927 11:22:24.637377 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:24 dist-test-slave-lsvg krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:22:24.758167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43115,127.0.98.61:46525,127.0.98.60:45755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:24.840893 22585 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 11:22:24.841133 22585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:24.841151 22585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:24.841172 22585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:24.842263 22585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:24.842296 22585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:24.842310 22585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:24.842324 22585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:24.842335 22585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:24.843734 22585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43115,127.0.98.61:46525,127.0.98.60:45755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:24.843940 22585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:24.844120 22585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:24.849140 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:24.849156 22593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:24.849136 22590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:24.849372 22585 server_base.cc:1034] running on GCE node
I20260927 11:22:24.849740 22585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:24.850013 22585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:24.851213 22585 hybrid_clock.cc:648] HybridClock initialized: now 1790508144851208 us; error 39 us; skew 500 ppm
Sep 27 11:22:24 dist-test-slave-lsvg krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:24.854009 22585 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:22:24.854480 22585 webserver.cc:460] Webserver started at http://127.0.98.62:44287/ using document root <none> and password file <none>
I20260927 11:22:24.854715 22585 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:24.854786 22585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:24.854939 22585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:24.856083 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "f229cbbdb11c44a5a20414d2c845efce"
format_stamp: "Formatted at 2026-09-27 11:22:24 on dist-test-slave-lsvg"
server_key: "04f99375dafb3b4fe4acda45fa058456"
server_key_iv: "9ad0ccfa0f234408bf7af795d9d44d57"
server_key_version: "encryptionkey@0"
I20260927 11:22:24.856462 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "f229cbbdb11c44a5a20414d2c845efce"
format_stamp: "Formatted at 2026-09-27 11:22:24 on dist-test-slave-lsvg"
server_key: "04f99375dafb3b4fe4acda45fa058456"
server_key_iv: "9ad0ccfa0f234408bf7af795d9d44d57"
server_key_version: "encryptionkey@0"
I20260927 11:22:24.858031 22585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:22:24.859036 22600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:24.859266 22585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:22:24.859426 22585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "f229cbbdb11c44a5a20414d2c845efce"
format_stamp: "Formatted at 2026-09-27 11:22:24 on dist-test-slave-lsvg"
server_key: "04f99375dafb3b4fe4acda45fa058456"
server_key_iv: "9ad0ccfa0f234408bf7af795d9d44d57"
server_key_version: "encryptionkey@0"
I20260927 11:22:24.859524 22585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:24.872700 22585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:24.875242 22585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:24.875434 22585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:24.879634 22585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43115
I20260927 11:22:24.879688 22652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43115 every 8 connection(s)
I20260927 11:22:24.880080 22585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:24.881224 22653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:24.882831 22653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:22:24.883373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 22585
I20260927 11:22:24.883499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:24.883682 392 external_mini_cluster.cc:1428] Setting key 2ed3b95ff0d11165ce86f06fd02fae7c
Sep 27 11:22:24 dist-test-slave-lsvg krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260927 11:22:24.894383 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:24.883604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43115 (local address 127.0.0.1:56450)
0927 11:22:24.883793 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:24.883849 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:24.884615 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:24.884916 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:24.884919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:24.885352 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:24.885527 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:24.885532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:24.886827 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:24.886829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:24.888022 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:24.888025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:24.888532 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:24.888533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:24.888630 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:24.889277 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:24.889290 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:24.891017 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:24.892920 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:24.892926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:24.892928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:24.893196 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:24.893402 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:24.893403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:24.893403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:24.893484 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:24.893806 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:24.893811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:24.893915 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 11:22:24.894020 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 11:22:24.894232 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":72,"thread_start_us":85,"threads_started":1}
I20260927 11:22:24.894392 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:24.883625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.0.98.62:43115)
0927 11:22:24.883920 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:24.883924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:24.883963 (+ 39us) server_negotiation.cc:407] Connection header received
0927 11:22:24.884773 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:24.884775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:24.884792 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:24.884872 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:24.885633 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:24.886716 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:24.888108 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:24.888412 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:24.888504 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:24.891126 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:24.891132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:24.891133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:24.891139 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:24.892835 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:24.893258 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:24.893261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:24.893262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:24.893342 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:24.893549 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:24.893550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:24.893550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:24.893751 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:24.893863 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:24.894049 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 11:22:24.894174 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":265,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 11:22:24.895212 22653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:22:24.895587 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:24.895446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46525 (local address 127.0.0.1:51420)
0927 11:22:24.895500 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:24.895552 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 11:22:24.895735 22601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46525: connect: Connection refused (error 111)
I20260927 11:22:24.896165 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:24.896054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46525 (local address 127.0.0.1:51436)
0927 11:22:24.896104 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:24.896134 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 11:22:24.896273 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46525: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46525: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 11:22:24.912351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46525
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43115,127.0.98.61:46525,127.0.98.60:45755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:22:24.945524 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } attempt: 1
I20260927 11:22:24.946031 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:24.945890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46525 (local address 127.0.0.1:51446)
0927 11:22:24.945958 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:24.946002 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 11:22:24.946384 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:24.946287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46525 (local address 127.0.0.1:51458)
0927 11:22:24.946329 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:24.946357 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 11:22:24.946482 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46525: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46525: connect: Connection refused (error 111)
W20260927 11:22:24.994949 22660 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 11:22:24.995230 22660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:24.995262 22660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:24.995303 22660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:24.996523 22660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:24.996603 22660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:24.996626 22660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:24.996647 22660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:24.996719 22660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:24.999632 22660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43115,127.0.98.61:46525,127.0.98.60:45755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:24.999974 22660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:25.000267 22660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:25.006780 22668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:25.006932 22660 server_base.cc:1034] running on GCE node
W20260927 11:22:25.006784 22665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:25.007113 22666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:25.007409 22660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:25.007694 22660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:25.008873 22660 hybrid_clock.cc:648] HybridClock initialized: now 1790508145008869 us; error 34 us; skew 500 ppm
Sep 27 11:22:25 dist-test-slave-lsvg krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:25.011562 22660 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 11:22:25.012053 22660 webserver.cc:460] Webserver started at http://127.0.98.61:33879/ using document root <none> and password file <none>
I20260927 11:22:25.012301 22660 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:25.012373 22660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:25.012529 22660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:25.013693 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/data/instance:
uuid: "844c301db2894b37ac873b9efdf283db"
format_stamp: "Formatted at 2026-09-27 11:22:25 on dist-test-slave-lsvg"
server_key: "595ceca51c880dc4ec794ed82232d74d"
server_key_iv: "5b41ab6391036a2cb2ea37c900e80642"
server_key_version: "encryptionkey@0"
I20260927 11:22:25.014070 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/wal/instance:
uuid: "844c301db2894b37ac873b9efdf283db"
format_stamp: "Formatted at 2026-09-27 11:22:25 on dist-test-slave-lsvg"
server_key: "595ceca51c880dc4ec794ed82232d74d"
server_key_iv: "5b41ab6391036a2cb2ea37c900e80642"
server_key_version: "encryptionkey@0"
I20260927 11:22:25.014609 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } attempt: 2
I20260927 11:22:25.014983 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.014863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46525 (local address 127.0.0.1:51460)
0927 11:22:25.014919 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.014956 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 11:22:25.015357 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.015226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46525 (local address 127.0.0.1:51472)
0927 11:22:25.015287 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.015324 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 11:22:25.015491 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46525: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46525: connect: Connection refused (error 111)
I20260927 11:22:25.015615 22660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:22:25.016299 22675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:25.016489 22660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:22:25.016564 22660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/wal
uuid: "844c301db2894b37ac873b9efdf283db"
format_stamp: "Formatted at 2026-09-27 11:22:25 on dist-test-slave-lsvg"
server_key: "595ceca51c880dc4ec794ed82232d74d"
server_key_iv: "5b41ab6391036a2cb2ea37c900e80642"
server_key_version: "encryptionkey@0"
I20260927 11:22:25.016657 22660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:25.038705 22660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:25.040913 22660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:25.041112 22660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:25.045193 22660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46525
I20260927 11:22:25.045269 22727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46525 every 8 connection(s)
I20260927 11:22:25.045619 22660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/data/info.pb
I20260927 11:22:25.046627 22728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:25.047734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 22660
I20260927 11:22:25.047835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-1/wal/instance
I20260927 11:22:25.048017 392 external_mini_cluster.cc:1428] Setting key 7376c68f36a227eec65364f20818fd67
I20260927 11:22:25.048574 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:22:25 dist-test-slave-lsvg krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 11:22:25.058992 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.049441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.0.98.62:43115)
0927 11:22:25.049492 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.049495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.049788 (+ 293us) server_negotiation.cc:407] Connection header received
0927 11:22:25.050251 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.050252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.050274 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.050361 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.051287 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.051776 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.053185 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.053401 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.053458 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.056004 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.056005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.056006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.056014 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.057503 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.057954 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.057956 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.057957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.058018 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.058304 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.058305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.058305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.058428 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.058506 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.058802 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.058891 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:22:25.059264 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.049443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43115 (local address 127.0.0.1:56452)
0927 11:22:25.049638 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.049681 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:25.050154 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:25.050468 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:25.050473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:25.050884 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:25.051179 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.051184 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.051881 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.051882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.053088 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.053091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.053511 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.053513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.053690 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.054260 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:25.054280 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:25.055898 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:25.057608 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.057617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.057619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.057862 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.058112 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.058112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.058113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.058227 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.058499 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:25.058506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:25.058665 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 11:22:25.058782 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 11:22:25.058954 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 11:22:25.059837 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:22:25 dist-test-slave-lsvg krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:22:25.068387 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.060127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46525 (local address 127.0.0.1:51474)
0927 11:22:25.060179 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.060189 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:25.060226 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:25.060499 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:25.060500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:25.060653 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:25.060704 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.060705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.062010 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.062011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.062535 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.062537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.063275 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.063276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.063333 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.063882 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:25.063887 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:25.065180 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:25.067357 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.067357 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.067358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.067520 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.067754 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.067755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.067755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.067805 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.068144 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:25.068144 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:25.068180 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 11:22:25.068227 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 11:22:25.068317 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 11:22:25.068460 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.060200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51474 (local address 127.0.98.61:46525)
0927 11:22:25.060364 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.060368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.060386 (+ 18us) server_negotiation.cc:407] Connection header received
0927 11:22:25.060403 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.060404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.060416 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.060499 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.060787 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.061944 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.062660 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.063190 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.063280 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.065266 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.065275 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.065275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.065283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.067271 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.067602 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.067606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.067606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.067679 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.067868 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.067869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.067870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.068074 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.068157 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.068264 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.068366 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260927 11:22:25.069145 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:22:25.069523 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.069400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45755 (local address 127.0.0.1:52044)
0927 11:22:25.069451 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.069502 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 11:22:25.069705 22677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
I20260927 11:22:25.070132 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.070043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45755 (local address 127.0.0.1:52052)
0927 11:22:25.070090 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.070115 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 11:22:25.070271 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45755: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 11:22:25.078002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45755
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43115,127.0.98.61:46525,127.0.98.60:45755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:22:25.106647 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } attempt: 3
Sep 27 11:22:25 dist-test-slave-lsvg krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:22:25.114359 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.107078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46525 (local address 127.0.0.1:51480)
0927 11:22:25.107143 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.107153 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:25.107212 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:25.107365 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:25.107366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:25.107537 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:25.107600 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.107602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.108608 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.108609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.109241 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.109243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.109680 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.109681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.109765 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.110270 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:25.110275 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:25.111494 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:25.113268 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.113269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.113269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.113448 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.113686 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.113687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.113687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.113753 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.114009 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:25.114010 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:25.114061 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 11:22:25.114151 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 11:22:25.114258 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 11:22:25.114359 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.107152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.0.98.61:46525)
0927 11:22:25.107222 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.107224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.107236 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:22:25.107266 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.107267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.107278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.107358 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.107705 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.108507 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.109333 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.109588 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.109674 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.111627 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.111628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.111628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.111634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.113169 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.113545 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.113547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.113548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.113608 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.113818 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.113819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.113819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.113939 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.114024 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.114177 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.114259 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:22:25.114953 22653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:22:25.115312 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.115207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45755 (local address 127.0.0.1:52068)
0927 11:22:25.115257 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.115285 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 11:22:25.115736 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.115648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45755 (local address 127.0.0.1:52078)
0927 11:22:25.115699 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.115722 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 11:22:25.115861 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45755: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
I20260927 11:22:25.119489 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } attempt: 1
I20260927 11:22:25.119871 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.119764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45755 (local address 127.0.0.1:52082)
0927 11:22:25.119817 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.119849 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 11:22:25.120184 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.120129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45755 (local address 127.0.0.1:52094)
0927 11:22:25.120150 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.120171 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260927 11:22:25.120292 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45755: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
I20260927 11:22:25.147032 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } attempt: 1
I20260927 11:22:25.147572 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.147414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45755 (local address 127.0.0.1:52110)
0927 11:22:25.147489 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.147541 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 11:22:25.148010 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.147867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45755 (local address 127.0.0.1:52122)
0927 11:22:25.147939 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.147978 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 11:22:25.148180 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45755: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
W20260927 11:22:25.163229 22735 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 11:22:25.163473 22735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:25.163492 22735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:25.163514 22735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:25.164666 22735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:25.164702 22735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:25.164716 22735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:25.164728 22735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:25.164740 22735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:25.166329 22735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43115,127.0.98.61:46525,127.0.98.60:45755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:25.166615 22735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:25.166827 22735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:25.171754 22740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:25.171743 22743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:25.171772 22735 server_base.cc:1034] running on GCE node
W20260927 11:22:25.171743 22741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:25.172263 22735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:25.172597 22735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:25.173736 22735 hybrid_clock.cc:648] HybridClock initialized: now 1790508145173716 us; error 36 us; skew 500 ppm
Sep 27 11:22:25 dist-test-slave-lsvg krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:25.176143 22735 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 11:22:25.176578 22735 webserver.cc:460] Webserver started at http://127.0.98.60:34639/ using document root <none> and password file <none>
I20260927 11:22:25.176822 22735 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:25.176893 22735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:25.177094 22735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:25.178174 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/data/instance:
uuid: "964b3c37e85644c984329aa405425c93"
format_stamp: "Formatted at 2026-09-27 11:22:25 on dist-test-slave-lsvg"
server_key: "2eb20441e5149d29a3417d35624f3c16"
server_key_iv: "72ba5f03b182fa41a8e578a8fc36cad2"
server_key_version: "encryptionkey@0"
I20260927 11:22:25.178543 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/wal/instance:
uuid: "964b3c37e85644c984329aa405425c93"
format_stamp: "Formatted at 2026-09-27 11:22:25 on dist-test-slave-lsvg"
server_key: "2eb20441e5149d29a3417d35624f3c16"
server_key_iv: "72ba5f03b182fa41a8e578a8fc36cad2"
server_key_version: "encryptionkey@0"
I20260927 11:22:25.179862 22735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:22:25.180621 22750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:25.180788 22735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:22:25.180891 22735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/wal
uuid: "964b3c37e85644c984329aa405425c93"
format_stamp: "Formatted at 2026-09-27 11:22:25 on dist-test-slave-lsvg"
server_key: "2eb20441e5149d29a3417d35624f3c16"
server_key_iv: "72ba5f03b182fa41a8e578a8fc36cad2"
server_key_version: "encryptionkey@0"
I20260927 11:22:25.180972 22735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:25.188441 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } attempt: 2
I20260927 11:22:25.188921 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.188783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45755 (local address 127.0.0.1:52136)
0927 11:22:25.188843 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.188889 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 11:22:25.189273 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.189202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45755 (local address 127.0.0.1:52140)
0927 11:22:25.189230 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.189254 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260927 11:22:25.189395 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45755: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45755: connect: Connection refused (error 111)
I20260927 11:22:25.196377 22735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:25.198536 22735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:25.198679 22735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:25.202857 22735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45755
I20260927 11:22:25.202941 22802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45755 every 8 connection(s)
I20260927 11:22:25.203315 22735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/data/info.pb
I20260927 11:22:25.204373 22803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:25.206071 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:22:25 dist-test-slave-lsvg krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:22:25.213260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 22735
I20260927 11:22:25.213374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507878147783-392-0/minicluster-data/master-2/wal/instance
I20260927 11:22:25.213590 392 external_mini_cluster.cc:1428] Setting key 04982e6bcf3eb703896b571f4865163c
I20260927 11:22:25.215556 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.206839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.0.98.62:43115)
0927 11:22:25.206918 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.206921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.207177 (+ 256us) server_negotiation.cc:407] Connection header received
0927 11:22:25.207627 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.207628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.207638 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.207700 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.208321 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.208795 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.209867 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.210058 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.210111 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.212259 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.212260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.212261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.212266 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.213865 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.214458 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.214462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.214462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.214539 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.214840 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.214840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.214841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.214949 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.215014 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.215372 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.215458 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:22:25.215692 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.206809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43115 (local address 127.0.0.1:56466)
0927 11:22:25.207049 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.207086 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:25.207537 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:25.207784 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:25.207788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:25.208051 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:25.208246 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.208251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.208880 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.208880 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.209793 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.209796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.210138 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.210139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.210246 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.210816 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:25.210824 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:25.212180 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:25.214010 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.214014 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.214016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.214377 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.214612 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.214613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.214614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.214769 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.215035 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:25.215041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:25.215228 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 11:22:25.215328 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 11:22:25.215464 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260927 11:22:25.216549 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:22:25 dist-test-slave-lsvg krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 11:22:25 dist-test-slave-lsvg krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:22:25.223349 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } attempt: 2
I20260927 11:22:25.224332 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.214011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.0.98.60:45755)
0927 11:22:25.214195 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.214200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.214220 (+ 20us) server_negotiation.cc:407] Connection header received
0927 11:22:25.214298 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.214302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.214313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.214381 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.215202 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.216192 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.217224 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.217625 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.217698 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.220587 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.220598 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.220599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.220607 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.222845 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.223219 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.223223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.223223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.223294 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.223689 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.223690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.223690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.223896 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.224072 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.224118 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.224202 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260927 11:22:25.224965 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.216913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.0.98.61:46525)
0927 11:22:25.216985 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.216987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.216995 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:22:25.217030 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.217031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.217041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.217115 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.217401 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.218088 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.219135 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.219414 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.219493 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.221604 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.221604 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.221605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.221612 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.223260 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.223636 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.223637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.223637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.223702 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.224113 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.224114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.224115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.224256 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.224302 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.224789 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.224871 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 11:22:25.225484 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.216842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46525 (local address 127.0.0.1:51494)
0927 11:22:25.216903 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.216913 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:25.216950 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:25.217125 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:25.217126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:25.217282 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:25.217331 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.217332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.218168 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.218170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.219056 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.219058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.219503 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.219505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.219585 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.219993 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:25.219997 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:25.221510 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:25.223353 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.223354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.223354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.223558 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.223955 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.223956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.223956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.224024 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.224355 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:25.224356 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:25.224663 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 11:22:25.225278 (+ 615us) client_negotiation.cc:240] Negotiation successful
0927 11:22:25.225374 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 11:22:25.226197 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:22:25 dist-test-slave-lsvg krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 11:22:25 dist-test-slave-lsvg krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 11:22:25 dist-test-slave-lsvg krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:22:25.233074 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.223692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45755 (local address 127.0.0.1:52156)
0927 11:22:25.223743 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.223756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:25.223796 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:25.224130 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:25.224132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:25.224297 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:25.224352 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.224354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.225460 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.225461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.226130 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.226133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.226978 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.226979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.227037 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.227454 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:25.227458 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:25.228887 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:25.231874 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.231875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.231875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.232101 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.232349 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.232350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.232350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.232419 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.232678 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:25.232679 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:25.232731 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 11:22:25.232815 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 11:22:25.232917 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 11:22:25.233538 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.223799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.0.98.60:45755)
0927 11:22:25.223982 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.223984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.223992 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:22:25.224017 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.224018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.224029 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.224425 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.224474 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.225361 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.226557 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.226886 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.226975 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.228977 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.228979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.228979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.228986 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.231783 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.232188 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.232189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.232190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.232254 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.232493 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.232494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.232494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.232605 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.232679 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.232854 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.233388 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":126,"threads_started":1}
I20260927 11:22:25.235325 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.226468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45755 (local address 127.0.0.1:52172)
0927 11:22:25.226525 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.226535 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:25.226571 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:25.226683 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:25.226684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:25.226827 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:25.226857 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.226858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.227705 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.227706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.228458 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.228461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.229984 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.229986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.230034 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.230410 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:25.230414 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:25.231945 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:25.234260 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.234260 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.234261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.234436 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.234656 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.234656 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.234656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.234709 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.234993 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:25.234994 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:25.235038 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 11:22:25.235120 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 11:22:25.235219 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 11:22:25.235386 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.226518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.0.98.60:45755)
0927 11:22:25.226552 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.226553 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.226562 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:22:25.226619 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.226620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.226628 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.226683 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.226915 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.227611 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.228539 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.229904 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.229980 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.232035 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.232036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.232036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.232044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.234172 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.234524 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.234525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.234526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.234576 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.234780 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.234781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.234781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.234915 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.234988 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.235142 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.235265 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":19}
I20260927 11:22:25.235437 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce: Bootstrap starting.
I20260927 11:22:25.236109 22653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:25.236335 22653 log.cc:824] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:25.236416 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93: Bootstrap starting.
I20260927 11:22:25.236891 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.225148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.0.98.62:43115)
0927 11:22:25.225188 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.225190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.225232 (+ 42us) server_negotiation.cc:407] Connection header received
0927 11:22:25.225303 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.225304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.225313 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.225368 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.226126 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.226793 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.227980 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.228182 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.228256 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.232458 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.232459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.232460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.232468 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.234194 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.234592 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.234593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.234594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.234646 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.234838 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.234839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.234839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.234961 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.235084 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.235308 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.235379 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:22:25.237021 22803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:25.237422 22803 log.cc:824] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:25.237468 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce: No bootstrap required, opened a new log
I20260927 11:22:25.239476 22653 raft_consensus.cc:348] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } }
I20260927 11:22:25.239624 22653 raft_consensus.cc:729] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f229cbbdb11c44a5a20414d2c845efce, State: Initialized, Role: FOLLOWER
I20260927 11:22:25.239692 22653 consensus_queue.cc:260] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } }
I20260927 11:22:25.239959 22811 sys_catalog.cc:455] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } } }
I20260927 11:22:25.240034 22653 sys_catalog.cc:564] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:25.240063 22811 sys_catalog.cc:458] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [sys.catalog]: This master's current role is: FOLLOWER
W20260927 11:22:25.241959 22822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:22:25.242027 22822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:22:25.242218 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93: No bootstrap required, opened a new log
Sep 27 11:22:25 dist-test-slave-lsvg krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:22:25.244432 22803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } }
I20260927 11:22:25.244593 22803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964b3c37e85644c984329aa405425c93, State: Initialized, Role: FOLLOWER
I20260927 11:22:25.244668 22803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } }
I20260927 11:22:25.244987 22803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:25.243114 22585 master_runner.cc:186] Error getting master registration for 127.0.98.61:46525: OK, instance_id { permanent_uuid: "844c301db2894b37ac873b9efdf283db" instance_seqno: 1790508145044016 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 11:22:25.246224 22585 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 11:22:25.246368 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } } }
I20260927 11:22:25.246492 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:22:25.246616 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.237705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51502 (local address 127.0.98.61:46525)
0927 11:22:25.237748 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.237750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.237761 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:22:25.237826 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.237827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.237835 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.237884 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.238472 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.238941 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.239860 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.240149 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.240221 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.243121 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.243123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.243124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.243132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.244906 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.245328 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.245330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.245330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.245387 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.245593 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.245594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.245594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.245731 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.245889 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.246430 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.246523 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 11:22:25.247529 22835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:22:25.247637 22835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:22:25.248430 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:25.249468 22627 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f229cbbdb11c44a5a20414d2c845efce"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56474
I20260927 11:22:25.249600 22627 raft_consensus.cc:482] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 11:22:25.249653 22627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:25.248138 22735 master_runner.cc:186] Error getting master registration for 127.0.98.61:46525: OK, instance_id { permanent_uuid: "844c301db2894b37ac873b9efdf283db" instance_seqno: 1790508145044016 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 11:22:25.250448 22735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 11:22:25.250581 22627 raft_consensus.cc:504] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } }
I20260927 11:22:25.250821 22627 leader_election.cc:288] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [CANDIDATE]: Term 1 election: Requested vote from peers 844c301db2894b37ac873b9efdf283db (127.0.98.61:46525), 964b3c37e85644c984329aa405425c93 (127.0.98.60:45755)
I20260927 11:22:25.251708 22777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f229cbbdb11c44a5a20414d2c845efce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "964b3c37e85644c984329aa405425c93"
I20260927 11:22:25.251824 22777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:25.252679 22777 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f229cbbdb11c44a5a20414d2c845efce in term 1.
I20260927 11:22:25.252885 22602 leader_election.cc:302] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 964b3c37e85644c984329aa405425c93, f229cbbdb11c44a5a20414d2c845efce; no voters:
I20260927 11:22:25.253034 22811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:25.253238 22811 raft_consensus.cc:686] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [term 1 LEADER]: Becoming Leader. State: Replica: f229cbbdb11c44a5a20414d2c845efce, State: Running, Role: LEADER
I20260927 11:22:25.253353 22811 consensus_queue.cc:237] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } }
I20260927 11:22:25.255424 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [sys.catalog]: SysCatalogTable state changed. Reason: New leader f229cbbdb11c44a5a20414d2c845efce. Latest consensus state: current_term: 1 leader_uuid: "f229cbbdb11c44a5a20414d2c845efce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } } }
I20260927 11:22:25.255602 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:25.255872 22845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:25.256102 22845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:25.266655 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.255054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.0.98.61:46525)
0927 11:22:25.255102 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.255104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.255985 (+ 881us) server_negotiation.cc:407] Connection header received
0927 11:22:25.256161 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.256162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.256189 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.256225 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.257554 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.258341 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.260371 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.260693 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.260737 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.263166 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.263168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.263168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.263176 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.264848 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.265526 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.265528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.265528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.265602 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.265962 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.265963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.265963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.266118 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.266198 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.266481 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.266559 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:22:25.269017 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.254851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.0.98.62:43115)
0927 11:22:25.255012 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.255014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.260166 (+ 5152us) server_negotiation.cc:407] Connection header received
0927 11:22:25.260264 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.260264 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.260274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.260303 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.261084 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.262160 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.263221 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.263820 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.263863 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.265673 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.265675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.265675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.265682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.267307 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.268167 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.268168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.268169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.268230 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.268439 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.268439 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.268440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.268592 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.268683 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.268821 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.268912 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260927 11:22:25.275677 22777 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93 [term 1 FOLLOWER]: Refusing update from remote peer f229cbbdb11c44a5a20414d2c845efce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:22:25.276099 22837 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [LEADER]: Connected to new peer: Peer: permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:22:25.277721 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f229cbbdb11c44a5a20414d2c845efce. Latest consensus state: current_term: 1 leader_uuid: "f229cbbdb11c44a5a20414d2c845efce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } } }
I20260927 11:22:25.277819 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:22:25.279271 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f229cbbdb11c44a5a20414d2c845efce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } } }
I20260927 11:22:25.279366 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:25.279570 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.259790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.0.98.60:45755)
0927 11:22:25.259831 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.259833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.265191 (+ 5358us) server_negotiation.cc:407] Connection header received
0927 11:22:25.265303 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.265304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.265317 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.265349 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.266067 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.266755 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.267686 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.267995 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.268055 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.274910 (+ 6855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.274912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.274913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.274934 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.276786 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.278364 (+ 1578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.278365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.278365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.278429 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.278787 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.278788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.278789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.278925 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.279015 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.279381 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.279478 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:22:25.280026 22824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f229cbbdb11c44a5a20414d2c845efce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } } }
I20260927 11:22:25.280112 22824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:22:25.280555 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } attempt: 3
I20260927 11:22:25.280696 22846 mvcc.cc:205] Tried to move back new op lower bound from 7333921363041280000 to 7333921362958622720. Current Snapshot: MvccSnapshot[applied={T|T < 7333921363041280000}]
I20260927 11:22:25.281172 22848 mvcc.cc:205] Tried to move back new op lower bound from 7333921363041280000 to 7333921362958622720. Current Snapshot: MvccSnapshot[applied={T|T < 7333921363041280000}]
I20260927 11:22:25.282426 22845 catalog_manager.cc:1347] Generated new cluster ID: c0d3a42592834ee1b10db092c78784be
I20260927 11:22:25.282475 22845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:22:25 dist-test-slave-lsvg krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:22:25.289403 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.280793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45755 (local address 127.0.0.1:52188)
0927 11:22:25.280840 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:25.280850 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:25.280885 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:25.281014 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:25.281014 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:25.281202 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:25.281252 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.281254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.282249 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.282251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.283092 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:25.283094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.283674 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.283676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:25.283749 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.284321 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:25.284326 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:25.285911 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:25.287998 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.287999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.287999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.288225 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.288521 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:25.288522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:25.288522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:25.288594 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:25.289078 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:25.289079 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:25.289140 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 11:22:25.289200 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 11:22:25.289300 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 11:22:25.289991 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.280836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.0.98.60:45755)
0927 11:22:25.280899 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.280901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.280908 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:22:25.280930 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.280931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.280940 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.281001 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.281359 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.282129 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.283202 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.283527 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.283587 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.286033 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:25.286034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:25.286035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:25.286042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:25.287873 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.288318 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.288320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.288320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.288407 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:25.288680 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:25.288680 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:25.288681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:25.288932 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:25.289584 (+ 652us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.289688 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.289796 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 11:22:25.291646 22728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db: Bootstrap starting.
I20260927 11:22:25.292413 22728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:25.292735 22728 log.cc:824] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:25.297771 22728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db: No bootstrap required, opened a new log
I20260927 11:22:25.299654 22845 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:25.299930 22728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } }
I20260927 11:22:25.300096 22728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844c301db2894b37ac873b9efdf283db, State: Initialized, Role: FOLLOWER
I20260927 11:22:25.300181 22728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } }
I20260927 11:22:25.300535 22728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:25.300701 22845 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:25.301427 22702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f229cbbdb11c44a5a20414d2c845efce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "844c301db2894b37ac873b9efdf283db"
I20260927 11:22:25.301558 22702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:25.302484 22702 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f229cbbdb11c44a5a20414d2c845efce in term 1.
I20260927 11:22:25.302882 22701 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db [term 1 FOLLOWER]: Refusing update from remote peer f229cbbdb11c44a5a20414d2c845efce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:22:25.303164 22837 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [LEADER]: Connected to new peer: Peer: permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:22:25.304925 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f229cbbdb11c44a5a20414d2c845efce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } } }
I20260927 11:22:25.305033 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:22:25.305341 22853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db [sys.catalog]: SysCatalogTable state changed. Reason: New leader f229cbbdb11c44a5a20414d2c845efce. Latest consensus state: current_term: 1 leader_uuid: "f229cbbdb11c44a5a20414d2c845efce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } } }
I20260927 11:22:25.305431 22853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:22:25.305181 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f229cbbdb11c44a5a20414d2c845efce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } } }
I20260927 11:22:25.307498 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:22:25.309764 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f229cbbdb11c44a5a20414d2c845efce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f229cbbdb11c44a5a20414d2c845efce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43115 } } peers { permanent_uuid: "844c301db2894b37ac873b9efdf283db" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46525 } } peers { permanent_uuid: "964b3c37e85644c984329aa405425c93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45755 } } }
I20260927 11:22:25.309861 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:25.310552 22845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f229cbbdb11c44a5a20414d2c845efce: Generated new TSK 0
I20260927 11:22:25.310737 22845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:22:25.312055 22870 catalog_manager.cc:1259] Loaded cluster ID: c0d3a42592834ee1b10db092c78784be
I20260927 11:22:25.312142 22870 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db: loading cluster ID for follower catalog manager: success
I20260927 11:22:25.313391 22870 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db: acquiring CA information for follower catalog manager: success
I20260927 11:22:25.313786 22870 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 844c301db2894b37ac873b9efdf283db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 11:22:25.339264 22656 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:25.337461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56502 (local address 127.0.98.62:43115)
0927 11:22:25.337523 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.337525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.338579 (+ 1054us) server_negotiation.cc:407] Connection header received
0927 11:22:25.338688 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.338689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.338701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.338732 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.339191 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56502: BlockingRecv error: recv got EOF from 127.0.0.1:56502 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 11:22:25.340255 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:25.337894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.0.98.60:45755)
0927 11:22:25.337938 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.337940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.339702 (+ 1762us) server_negotiation.cc:407] Connection header received
0927 11:22:25.339801 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.339802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.339813 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.339842 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.340191 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52192: BlockingRecv error: recv got EOF from 127.0.0.1:52192 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 11:22:25.341190 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:25.338293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51516 (local address 127.0.98.61:46525)
0927 11:22:25.338335 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.338337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.340617 (+ 2280us) server_negotiation.cc:407] Connection header received
0927 11:22:25.340719 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.340720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.340731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.340761 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.341123 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51516: BlockingRecv error: recv got EOF from 127.0.0.1:51516 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:22:25.346524 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.342946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.0.98.61:46525)
0927 11:22:25.342982 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.342984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.343234 (+ 250us) server_negotiation.cc:407] Connection header received
0927 11:22:25.343343 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.343344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.343387 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.343428 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:25.343640 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.344369 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.345548 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.345828 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.345870 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.346103 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:25.346246 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:25.346299 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.346416 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.346438 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 11:22:25.350203 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:25.349314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.0.98.60:45755)
0927 11:22:25.349354 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.349356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.349578 (+ 222us) server_negotiation.cc:407] Connection header received
0927 11:22:25.349692 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.349694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.349714 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.349760 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.350145 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52200: BlockingRecv error: recv got EOF from 127.0.0.1:52200 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:22:25.355988 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.352665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.0.98.61:46525)
0927 11:22:25.352703 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.352705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.352928 (+ 223us) server_negotiation.cc:407] Connection header received
0927 11:22:25.353024 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.353025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.353036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.353164 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:25.353359 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.354064 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.355150 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.355430 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.355476 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.355718 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:25.355756 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:25.355784 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.355893 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.355915 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 11:22:25.359611 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:25.358620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52212 (local address 127.0.98.60:45755)
0927 11:22:25.358662 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.358664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.359028 (+ 364us) server_negotiation.cc:407] Connection header received
0927 11:22:25.359132 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.359133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.359144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.359174 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.359555 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52212: BlockingRecv error: recv got EOF from 127.0.0.1:52212 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:22:25.366358 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.363025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.0.98.61:46525)
0927 11:22:25.363064 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.363066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.363292 (+ 226us) server_negotiation.cc:407] Connection header received
0927 11:22:25.363393 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.363393 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.363405 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.363434 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:25.363649 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.364407 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.365572 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.365852 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.365893 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.366095 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:25.366128 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:25.366156 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.366263 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.366281 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 11:22:25.369868 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:25.368984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.0.98.60:45755)
0927 11:22:25.369026 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.369028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.369291 (+ 263us) server_negotiation.cc:407] Connection header received
0927 11:22:25.369396 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.369397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.369410 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.369439 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.369809 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41020: BlockingRecv error: recv got EOF from 127.0.0.1:41020 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:22:25.378698 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.375307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.0.98.61:46525)
0927 11:22:25.375354 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.375356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.375630 (+ 274us) server_negotiation.cc:407] Connection header received
0927 11:22:25.375737 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.375738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.375749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.375779 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:25.375967 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.376661 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.377834 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.378136 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.378183 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.378408 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:25.378443 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:25.378471 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.378597 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.378618 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 11:22:25.382169 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:25.381303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.0.98.60:45755)
0927 11:22:25.381348 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.381350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.381619 (+ 269us) server_negotiation.cc:407] Connection header received
0927 11:22:25.381720 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.381721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.381732 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.381761 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.382116 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41034: BlockingRecv error: recv got EOF from 127.0.0.1:41034 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:22:25.394976 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.391634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57226 (local address 127.0.98.61:46525)
0927 11:22:25.391676 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.391678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.391926 (+ 248us) server_negotiation.cc:407] Connection header received
0927 11:22:25.392037 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.392038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.392050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.392079 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:25.392271 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.393005 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.394170 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.394423 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.394465 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.394702 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:25.394737 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:25.394765 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.394870 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.394890 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 11:22:25.398366 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:25.397478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.0.98.60:45755)
0927 11:22:25.397530 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.397532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.397783 (+ 251us) server_negotiation.cc:407] Connection header received
0927 11:22:25.397881 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.397882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.397896 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.397926 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.398307 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41040: BlockingRecv error: recv got EOF from 127.0.0.1:41040 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:22:25.419607 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.415929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.0.98.61:46525)
0927 11:22:25.415979 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.415982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.416247 (+ 265us) server_negotiation.cc:407] Connection header received
0927 11:22:25.416351 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.416352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.416362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.416391 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:25.416584 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.417417 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.418680 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.418993 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.419058 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.419329 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:25.419363 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:25.419391 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.419497 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.419532 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 11:22:25.422952 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:25.422189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.0.98.60:45755)
0927 11:22:25.422232 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.422234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.422411 (+ 177us) server_negotiation.cc:407] Connection header received
0927 11:22:25.422512 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.422513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.422525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.422554 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.422892 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41042: BlockingRecv error: recv got EOF from 127.0.0.1:41042 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:22:25.459046 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.456410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.0.98.61:46525)
0927 11:22:25.456460 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.456462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.456624 (+ 162us) server_negotiation.cc:407] Connection header received
0927 11:22:25.456696 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.456697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.456709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.456776 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:25.456927 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.457482 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.458297 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.458497 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.458593 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.458756 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:25.458796 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:25.458855 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.458916 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.458938 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 11:22:25.461794 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:25.461115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41048 (local address 127.0.98.60:45755)
0927 11:22:25.461162 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.461164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.461266 (+ 102us) server_negotiation.cc:407] Connection header received
0927 11:22:25.461334 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.461335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.461347 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.461406 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.461736 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41048: BlockingRecv error: recv got EOF from 127.0.0.1:41048 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:22:25.529347 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.526986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.0.98.61:46525)
0927 11:22:25.527100 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.527102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.527151 (+ 49us) server_negotiation.cc:407] Connection header received
0927 11:22:25.527216 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.527217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.527227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.527281 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:25.527413 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.527890 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.528647 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.528879 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.528951 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.529091 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:25.529130 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:25.529169 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.529231 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.529248 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 11:22:25.531646 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:25.531010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.0.98.60:45755)
0927 11:22:25.531059 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.531061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.531073 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:22:25.531138 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.531138 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.531148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.531194 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.531591 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41054: BlockingRecv error: recv got EOF from 127.0.0.1:41054 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:22:25.663519 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.660921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.0.98.61:46525)
0927 11:22:25.661013 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.661015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.661135 (+ 120us) server_negotiation.cc:407] Connection header received
0927 11:22:25.661236 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.661237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.661249 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.661324 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:25.661476 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.661958 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.662803 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.663003 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.663079 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.663223 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:25.663263 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:25.663323 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.663401 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.663421 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 11:22:25.665973 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:25.665264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.0.98.60:45755)
0927 11:22:25.665333 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.665335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.665424 (+ 89us) server_negotiation.cc:407] Connection header received
0927 11:22:25.665510 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.665511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.665522 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.665589 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.665912 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41056: BlockingRecv error: recv got EOF from 127.0.0.1:41056 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 11:22:25.925302 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:25.923144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57246 (local address 127.0.98.61:46525)
0927 11:22:25.923221 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.923223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.923251 (+ 28us) server_negotiation.cc:407] Connection header received
0927 11:22:25.923332 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.923333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.923341 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.923385 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:25.923519 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.924003 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.924717 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:25.924893 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:25.924940 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:25.925105 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:25.925131 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:25.925170 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:25.925228 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 11:22:25.925242 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 11:22:25.927634 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:25.927085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41058 (local address 127.0.98.60:45755)
0927 11:22:25.927151 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:25.927154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:25.927163 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:22:25.927190 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:25.927191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:25.927202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:25.927253 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:25.927568 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41058: BlockingRecv error: recv got EOF from 127.0.0.1:41058 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:22:26.248136 22835 catalog_manager.cc:1259] Loaded cluster ID: c0d3a42592834ee1b10db092c78784be
I20260927 11:22:26.248214 22835 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93: loading cluster ID for follower catalog manager: success
I20260927 11:22:26.249091 22835 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93: acquiring CA information for follower catalog manager: success
I20260927 11:22:26.249470 22835 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 964b3c37e85644c984329aa405425c93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 11:22:26.443154 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:26.440740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.0.98.61:46525)
0927 11:22:26.440924 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:26.440926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:26.440939 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:22:26.441041 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:26.441041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:26.441083 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:26.441166 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:26.441277 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:26.441768 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:26.442527 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:26.442721 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:26.442777 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:26.442891 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:26.442919 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:26.442957 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:26.443017 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 11:22:26.443035 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260927 11:22:26.447350 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:26.444868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.0.98.60:45755)
0927 11:22:26.444972 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:26.444974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:26.444982 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:22:26.445024 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:26.445024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:26.445125 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:26.445195 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:26.445285 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:26.445897 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:26.446609 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:26.446839 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:26.446890 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:26.447017 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:26.447119 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:26.447177 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:26.447191 (+ 14us) server_negotiation.cc:299] Negotiation successful
0927 11:22:26.447214 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":50,"threads_started":1}
I20260927 11:22:26.451984 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:26.449037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.0.98.60:45755)
0927 11:22:26.449121 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:26.449123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:26.449161 (+ 38us) server_negotiation.cc:407] Connection header received
0927 11:22:26.449226 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:26.449227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:26.449236 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:26.449287 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:26.449420 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:26.450105 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:26.451285 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:26.451488 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:26.451570 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:26.451717 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:26.451749 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:26.451783 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:26.451894 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 11:22:26.451915 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 11:22:26.452051 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:26.449012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.0.98.61:46525)
0927 11:22:26.449079 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:26.449081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:26.449103 (+ 22us) server_negotiation.cc:407] Connection header received
0927 11:22:26.449239 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:26.449240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:26.449252 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:26.449281 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:26.449474 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:26.450034 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:26.451264 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:26.451557 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:26.451619 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:26.451778 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:26.451811 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:26.451858 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:26.451913 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 11:22:26.451935 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 11:22:26.452359 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:26.448933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.0.98.62:43115)
0927 11:22:26.449115 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:26.449118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:26.449420 (+ 302us) server_negotiation.cc:407] Connection header received
0927 11:22:26.449482 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:26.449483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:26.449533 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:26.449619 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:26.449753 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:26.450388 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:26.451300 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:26.451605 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:26.451676 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:26.451886 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:26.451998 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:26.452058 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:26.452281 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 11:22:26.452304 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":49,"threads_started":1}
I20260927 11:22:26.453992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 22585
I20260927 11:22:26.459456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 22660
I20260927 11:22:26.464165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 22735
2026-09-27T11:22:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2869 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:22:26 dist-test-slave-lsvg krb5kdc[23003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:22:26 dist-test-slave-lsvg krb5kdc[23003](info): set up 2 sockets
Sep 27 11:22:26 dist-test-slave-lsvg krb5kdc[23003](info): commencing operation
krb5kdc: starting...
W20260927 11:22:27.513835 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/test-user.keytab.
Sep 27 11:22:27 dist-test-slave-lsvg krb5kdc[23003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:22:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:22:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:22:27.637179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43417
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43417,127.0.98.61:46131,127.0.98.60:38345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:22:27.717905 23016 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 11:22:27.718142 23016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:27.718161 23016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:27.718181 23016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:27.719272 23016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:27.719309 23016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:27.719324 23016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:27.719336 23016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:27.719349 23016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:27.720788 23016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43417,127.0.98.61:46131,127.0.98.60:38345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:27.720985 23016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:27.721249 23016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:27.726245 23024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:27.726214 23021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:27.726217 23022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:27.726382 23016 server_base.cc:1034] running on GCE node
I20260927 11:22:27.726683 23016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:27.727010 23016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:27.728186 23016 hybrid_clock.cc:648] HybridClock initialized: now 1790508147728163 us; error 40 us; skew 500 ppm
Sep 27 11:22:27 dist-test-slave-lsvg krb5kdc[23003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:27.730754 23016 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:22:27.731210 23016 webserver.cc:460] Webserver started at http://127.0.98.62:41347/ using document root <none> and password file <none>
I20260927 11:22:27.731456 23016 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:27.731530 23016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:27.731678 23016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:27.732744 23016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/data/instance:
uuid: "cef535dad7d0467d9ca6371c66e5bb78"
format_stamp: "Formatted at 2026-09-27 11:22:27 on dist-test-slave-lsvg"
server_key: "243184484c745031692f9b3086f9f768"
server_key_iv: "36dfb15d3c680173854cc32bc041d948"
server_key_version: "encryptionkey@0"
I20260927 11:22:27.733196 23016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/wal/instance:
uuid: "cef535dad7d0467d9ca6371c66e5bb78"
format_stamp: "Formatted at 2026-09-27 11:22:27 on dist-test-slave-lsvg"
server_key: "243184484c745031692f9b3086f9f768"
server_key_iv: "36dfb15d3c680173854cc32bc041d948"
server_key_version: "encryptionkey@0"
I20260927 11:22:27.734557 23016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:22:27.735261 23031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:27.735424 23016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:22:27.735524 23016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/wal
uuid: "cef535dad7d0467d9ca6371c66e5bb78"
format_stamp: "Formatted at 2026-09-27 11:22:27 on dist-test-slave-lsvg"
server_key: "243184484c745031692f9b3086f9f768"
server_key_iv: "36dfb15d3c680173854cc32bc041d948"
server_key_version: "encryptionkey@0"
I20260927 11:22:27.735608 23016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:27.752486 23016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:27.754956 23016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:27.755125 23016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:27.759291 23016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43417
I20260927 11:22:27.759327 23083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43417 every 8 connection(s)
I20260927 11:22:27.759685 23016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:22:27.760704 23084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:27.762077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 23016
I20260927 11:22:27.762195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-0/wal/instance
I20260927 11:22:27.762413 392 external_mini_cluster.cc:1428] Setting key 0e1bae62665e7a1b4305b11aacd3dd42
I20260927 11:22:27.762447 23084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:22:27 dist-test-slave-lsvg krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 11:22:27.772477 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.763279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43417 (local address 127.0.0.1:37390)
0927 11:22:27.763460 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.763496 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:27.764025 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:27.764324 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:27.764327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:27.764619 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:27.764775 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:27.764780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.765891 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.765892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:27.766574 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:27.766576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.767087 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.767088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:27.767153 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:27.767737 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:27.767745 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:27.769151 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:27.770979 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:27.770985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:27.770987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:27.771179 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:27.771430 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:27.771431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:27.771431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:27.771533 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:27.771881 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:27.771885 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:27.771948 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 11:22:27.772059 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 11:22:27.772284 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":86,"thread_start_us":93,"threads_started":1}
I20260927 11:22:27.772481 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.763496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.0.98.62:43417)
0927 11:22:27.763620 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:27.763624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:27.763657 (+ 33us) server_negotiation.cc:407] Connection header received
0927 11:22:27.764199 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:27.764201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:27.764218 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:27.764320 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:27.764888 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.765773 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.766675 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.766981 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.767105 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:27.769248 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:27.769256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:27.769256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:27.769263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:27.770864 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:27.771278 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:27.771281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:27.771282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:27.771360 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:27.771594 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:27.771595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:27.771595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:27.771803 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:27.771893 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:27.772083 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 11:22:27.772223 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
I20260927 11:22:27.773392 23084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:22:27.773775 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.773635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46131 (local address 127.0.0.1:43304)
0927 11:22:27.773685 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.773738 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 11:22:27.773901 23034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46131: connect: Connection refused (error 111)
I20260927 11:22:27.774363 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.774249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46131 (local address 127.0.0.1:43314)
0927 11:22:27.774296 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.774331 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 11:22:27.774554 23084 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46131: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46131: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 11:22:27.788800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46131
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43417,127.0.98.61:46131,127.0.98.60:38345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:22:27.823820 23084 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } attempt: 1
I20260927 11:22:27.824375 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.824216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46131 (local address 127.0.0.1:43318)
0927 11:22:27.824299 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.824344 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 11:22:27.824761 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.824656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46131 (local address 127.0.0.1:43322)
0927 11:22:27.824706 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.824735 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 11:22:27.824872 23084 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46131: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46131: connect: Connection refused (error 111)
W20260927 11:22:27.870719 23091 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 11:22:27.870971 23091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:27.870990 23091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:27.871012 23091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:22:27.872298 23091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:27.872351 23091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:27.872366 23091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:27.872380 23091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:27.872392 23091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:27.874303 23091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43417,127.0.98.61:46131,127.0.98.60:38345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:27.874617 23091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:27.874886 23091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:27.880157 23096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:27.880154 23099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:27.880154 23097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:27.880242 23091 server_base.cc:1034] running on GCE node
I20260927 11:22:27.880618 23091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:27.880914 23091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:27.882085 23091 hybrid_clock.cc:648] HybridClock initialized: now 1790508147882073 us; error 35 us; skew 500 ppm
Sep 27 11:22:27 dist-test-slave-lsvg krb5kdc[23003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:27.884655 23091 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 11:22:27.885190 23091 webserver.cc:460] Webserver started at http://127.0.98.61:43933/ using document root <none> and password file <none>
I20260927 11:22:27.885385 23091 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:27.885429 23091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:27.885527 23091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:27.886549 23091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/data/instance:
uuid: "dc69efcbddef4ed387f79e44e33bbeb0"
format_stamp: "Formatted at 2026-09-27 11:22:27 on dist-test-slave-lsvg"
server_key: "6bd30a8191ad3bbdca4de5fd642be5a7"
server_key_iv: "81aee5a40e3a084af87c5292c4e5d86e"
server_key_version: "encryptionkey@0"
I20260927 11:22:27.886853 23091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/wal/instance:
uuid: "dc69efcbddef4ed387f79e44e33bbeb0"
format_stamp: "Formatted at 2026-09-27 11:22:27 on dist-test-slave-lsvg"
server_key: "6bd30a8191ad3bbdca4de5fd642be5a7"
server_key_iv: "81aee5a40e3a084af87c5292c4e5d86e"
server_key_version: "encryptionkey@0"
I20260927 11:22:27.888230 23091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:22:27.888942 23106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:27.889103 23091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:22:27.889195 23091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/wal
uuid: "dc69efcbddef4ed387f79e44e33bbeb0"
format_stamp: "Formatted at 2026-09-27 11:22:27 on dist-test-slave-lsvg"
server_key: "6bd30a8191ad3bbdca4de5fd642be5a7"
server_key_iv: "81aee5a40e3a084af87c5292c4e5d86e"
server_key_version: "encryptionkey@0"
I20260927 11:22:27.889251 23091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:27.893126 23084 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } attempt: 2
I20260927 11:22:27.893608 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.893457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46131 (local address 127.0.0.1:43330)
0927 11:22:27.893528 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.893572 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 11:22:27.894059 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.893934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46131 (local address 127.0.0.1:43344)
0927 11:22:27.894001 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.894033 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 11:22:27.894191 23084 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46131: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46131: connect: Connection refused (error 111)
I20260927 11:22:27.902504 23091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:27.904565 23091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:27.904727 23091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:27.908768 23091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46131
I20260927 11:22:27.908802 23158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46131 every 8 connection(s)
I20260927 11:22:27.909256 23091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/data/info.pb
I20260927 11:22:27.910310 23159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:27.911931 23159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:22:27.913795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 23091
I20260927 11:22:27.913919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-1/wal/instance
I20260927 11:22:27.914126 392 external_mini_cluster.cc:1428] Setting key 41f920abbb871197e067cfd74e01cf8d
Sep 27 11:22:27 dist-test-slave-lsvg krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 11:22:27.922050 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.912716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.0.98.62:43417)
0927 11:22:27.912776 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:27.912778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:27.913018 (+ 240us) server_negotiation.cc:407] Connection header received
0927 11:22:27.913578 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:27.913580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:27.913594 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:27.913631 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:27.914442 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.914921 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.916507 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.916720 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.916826 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:27.919109 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:27.919111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:27.919111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:27.919119 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:27.920563 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:27.921013 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:27.921014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:27.921015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:27.921125 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:27.921394 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:27.921395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:27.921395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:27.921521 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:27.921598 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:27.921875 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 11:22:27.921957 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 11:22:27.922250 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.912684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43417 (local address 127.0.0.1:37394)
0927 11:22:27.912875 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.912918 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:27.913441 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:27.913720 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:27.913724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:27.914080 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:27.914334 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:27.914340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.915058 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.915059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:27.916412 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:27.916415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.916848 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.916850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:27.916989 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:27.917602 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:27.917612 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:27.919007 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:27.920667 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:27.920675 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:27.920677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:27.920900 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:27.921193 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:27.921194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:27.921195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:27.921311 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:27.921589 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:27.921596 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:27.921739 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 11:22:27.921844 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 11:22:27.922010 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260927 11:22:27.922848 23159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 11:22:27 dist-test-slave-lsvg krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:22:27.931803 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.923117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46131 (local address 127.0.0.1:43360)
0927 11:22:27.923317 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.923325 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:27.923362 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:27.923540 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:27.923541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:27.923700 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:27.923759 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:27.923761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.925002 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.925003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:27.925892 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:27.925895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.926528 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.926529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:27.926640 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:27.927166 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:27.927171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:27.928493 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:27.930650 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:27.930651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:27.930651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:27.930814 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:27.931056 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:27.931057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:27.931057 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:27.931118 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:27.931539 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:27.931544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:27.931583 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 11:22:27.931645 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 11:22:27.931735 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260927 11:22:27.931946 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.923182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43360 (local address 127.0.98.61:46131)
0927 11:22:27.923350 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:27.923354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:27.923391 (+ 37us) server_negotiation.cc:407] Connection header received
0927 11:22:27.923415 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:27.923417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:27.923432 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:27.923511 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:27.923822 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.924906 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.925981 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.926410 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.926498 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:27.928579 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:27.928589 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:27.928589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:27.928598 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:27.930558 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:27.930901 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:27.930904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:27.930905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:27.930965 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:27.931204 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:27.931205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:27.931205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:27.931417 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:27.931548 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:27.931708 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 11:22:27.931799 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260927 11:22:27.932750 23159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:22:27.933128 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.932982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38345 (local address 127.0.0.1:39984)
0927 11:22:27.933016 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.933107 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260927 11:22:27.933277 23110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
I20260927 11:22:27.933692 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.933604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38345 (local address 127.0.0.1:40000)
0927 11:22:27.933651 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.933675 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 11:22:27.933810 23159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 11:22:27.940075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuJOgQn/build/release/bin/kudu
/tmp/dist-test-taskuJOgQn/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38345
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43417,127.0.98.61:46131,127.0.98.60:38345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:22:27.983095 23159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } attempt: 1
I20260927 11:22:27.983844 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.983638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38345 (local address 127.0.0.1:40004)
0927 11:22:27.983735 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.983782 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 11:22:27.984236 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.984128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38345 (local address 127.0.0.1:40016)
0927 11:22:27.984180 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.984205 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 11:22:27.984328 23159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
I20260927 11:22:27.985368 23084 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } attempt: 3
Sep 27 11:22:27 dist-test-slave-lsvg krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:22:27.992895 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.985659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46131 (local address 127.0.0.1:43366)
0927 11:22:27.985709 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.985719 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:27.985760 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:27.985911 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:27.985912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:27.986093 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:27.986175 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:27.986177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.986827 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.986828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:27.987702 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:27.987706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.988117 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.988119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:27.988202 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:27.988701 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:27.988707 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:27.990083 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:27.991803 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:27.991804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:27.991805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:27.991994 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:27.992210 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:27.992211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:27.992211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:27.992276 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:27.992546 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:27.992547 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:27.992596 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 11:22:27.992688 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 11:22:27.992791 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 11:22:27.992908 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.985713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.0.98.61:46131)
0927 11:22:27.985749 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:27.985751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:27.985769 (+ 18us) server_negotiation.cc:407] Connection header received
0927 11:22:27.985815 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:27.985816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:27.985825 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:27.985878 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:27.986249 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.986735 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.987818 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:27.988033 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:27.988100 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:27.990183 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:27.990185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:27.990185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:27.990194 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:27.991702 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:27.992084 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:27.992086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:27.992086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:27.992142 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:27.992342 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:27.992342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:27.992343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:27.992463 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:27.992542 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:27.992713 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 11:22:27.992791 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:22:27.993337 23084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:22:27.993754 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.993625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38345 (local address 127.0.0.1:40022)
0927 11:22:27.993695 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.993729 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 11:22:27.994127 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:27.994050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38345 (local address 127.0.0.1:40032)
0927 11:22:27.994080 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:27.994104 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260927 11:22:27.994225 23084 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
W20260927 11:22:28.024376 23166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 11:22:28.024642 23166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:22:28.024672 23166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:22:28.024715 23166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 11:22:28.025372 23084 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } attempt: 1
I20260927 11:22:28.025838 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.025704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38345 (local address 127.0.0.1:40034)
0927 11:22:28.025772 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:28.025809 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 11:22:28.026012 23166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:22:28.026074 23166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:22:28.026101 23166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:22:28.026130 23166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:22:28.026186 23166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:22:28.026211 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.026126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38345 (local address 127.0.0.1:40040)
0927 11:22:28.026157 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:28.026183 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260927 11:22:28.026310 23084 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
I20260927 11:22:28.027778 23166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43417,127.0.98.61:46131,127.0.98.60:38345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:22:28.028047 23166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:22:28.028294 23166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:22:28.033303 23172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:28.033313 23171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:22:28.033317 23174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:22:28.033347 23166 server_base.cc:1034] running on GCE node
I20260927 11:22:28.033752 23166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:22:28.034020 23166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:22:28.035162 23166 hybrid_clock.cc:648] HybridClock initialized: now 1790508148035150 us; error 30 us; skew 500 ppm
Sep 27 11:22:28 dist-test-slave-lsvg krb5kdc[23003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:22:28.037853 23166 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 11:22:28.038266 23166 webserver.cc:460] Webserver started at http://127.0.98.60:41247/ using document root <none> and password file <none>
I20260927 11:22:28.038543 23166 fs_manager.cc:359] Metadata directory not provided
I20260927 11:22:28.038594 23166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:22:28.038726 23166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:22:28.039837 23166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/data/instance:
uuid: "43c63758a96f4a44b074f064f949f589"
format_stamp: "Formatted at 2026-09-27 11:22:28 on dist-test-slave-lsvg"
server_key: "cc271efb37ea0cd3a11c60a62391e2b3"
server_key_iv: "fa131476e2d16563379ab94aacf3f2c6"
server_key_version: "encryptionkey@0"
I20260927 11:22:28.040205 23166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/wal/instance:
uuid: "43c63758a96f4a44b074f064f949f589"
format_stamp: "Formatted at 2026-09-27 11:22:28 on dist-test-slave-lsvg"
server_key: "cc271efb37ea0cd3a11c60a62391e2b3"
server_key_iv: "fa131476e2d16563379ab94aacf3f2c6"
server_key_version: "encryptionkey@0"
I20260927 11:22:28.041630 23166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260927 11:22:28.042443 23181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:22:28.042661 23166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:22:28.042755 23166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/wal
uuid: "43c63758a96f4a44b074f064f949f589"
format_stamp: "Formatted at 2026-09-27 11:22:28 on dist-test-slave-lsvg"
server_key: "cc271efb37ea0cd3a11c60a62391e2b3"
server_key_iv: "fa131476e2d16563379ab94aacf3f2c6"
server_key_version: "encryptionkey@0"
I20260927 11:22:28.042831 23166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:22:28.052495 23159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } attempt: 2
I20260927 11:22:28.053021 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.052872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38345 (local address 127.0.0.1:40048)
0927 11:22:28.052945 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:28.052991 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 11:22:28.053476 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.053368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38345 (local address 127.0.0.1:40060)
0927 11:22:28.053417 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:28.053444 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 11:22:28.053570 23159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38345: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38345: connect: Connection refused (error 111)
I20260927 11:22:28.060909 23166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:22:28.063515 23166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:22:28.063691 23166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:22:28.068014 23166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38345
I20260927 11:22:28.068099 23233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38345 every 8 connection(s)
I20260927 11:22:28.068470 23166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/data/info.pb
I20260927 11:22:28.069559 23234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:22:28.071169 23234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:22:28.075260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuJOgQn/build/release/bin/kudu as pid 23166
I20260927 11:22:28.075366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuJOgQn/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507878147783-392-0/minicluster-data/master-2/wal/instance
I20260927 11:22:28.075580 392 external_mini_cluster.cc:1428] Setting key e60d34d11dc026f98b364a8c09bbc899
Sep 27 11:22:28 dist-test-slave-lsvg krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:22:28.080968 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.071952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.0.98.62:43417)
0927 11:22:28.072039 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.072042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.072234 (+ 192us) server_negotiation.cc:407] Connection header received
0927 11:22:28.072767 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.072768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.072781 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.072843 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.073583 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.074065 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.075104 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.075338 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.075411 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.077860 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:28.077862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:28.077863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:28.077870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:28.079426 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.079841 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.079842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.079843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.079928 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.080260 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.080261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.080262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.080392 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:28.080450 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:28.080836 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 11:22:28.080901 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 11:22:28.081316 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.071906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43417 (local address 127.0.0.1:37408)
0927 11:22:28.072106 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:28.072143 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:28.072648 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:28.072924 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:28.072927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:28.073270 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:28.073478 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:28.073483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.074169 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.074170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:28.075005 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:28.075007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.075448 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.075449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:28.075616 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.076303 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:28.076313 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:28.077758 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:28.079519 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:28.079526 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:28.079528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:28.079761 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:28.080002 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:28.080003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:28.080004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:28.080177 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:28.080469 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:28.080476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:28.080677 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 11:22:28.080834 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 11:22:28.081011 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
Sep 27 11:22:28 dist-test-slave-lsvg krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:22:28.081988 23234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:22:28.085466 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.075995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.0.98.60:38345)
0927 11:22:28.076187 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.076192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.076250 (+ 58us) server_negotiation.cc:407] Connection header received
0927 11:22:28.076952 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.076958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.076984 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.077098 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.077749 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.078432 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.079262 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.079582 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.079676 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.082496 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:28.082505 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:28.082505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:28.082512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:28.084113 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.084545 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.084548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.084548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.084608 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.084827 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.084828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.084828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.084972 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:28.085150 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:28.085248 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 11:22:28.085347 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":129,"threads_started":1}
Sep 27 11:22:28 dist-test-slave-lsvg krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:22:28.089852 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.082248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46131 (local address 127.0.0.1:43368)
0927 11:22:28.082296 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:28.082306 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:28.082341 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:28.082534 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:28.082535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:28.082673 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:28.082754 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:28.082756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.083742 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.083743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:28.084214 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:28.084216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.084634 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.084635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:28.084706 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.085145 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:28.085150 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:28.086872 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:28.088788 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:28.088789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:28.088790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:28.088958 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:28.089198 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:28.089199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:28.089200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:28.089257 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:28.089528 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:28.089529 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:28.089571 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 11:22:28.089659 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 11:22:28.089756 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 11:22:28.089856 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.082336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.0.98.61:46131)
0927 11:22:28.082409 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.082411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.082419 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:22:28.082444 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.082445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.082462 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.082523 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.082847 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.083653 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.084333 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.084558 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.084638 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.086971 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:28.086973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:28.086973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:28.086980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:28.088682 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.089066 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.089067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.089067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.089125 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.089347 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.089348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.089348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.089461 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:28.089537 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:28.089686 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 11:22:28.089761 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:22:28.090236 23234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:22:28 dist-test-slave-lsvg krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 11:22:28 dist-test-slave-lsvg krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:22:28.094564 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.086671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.0.98.62:43417)
0927 11:22:28.086728 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.086730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.086736 (+ 6us) server_negotiation.cc:407] Connection header received
0927 11:22:28.086755 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.086756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.086763 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.086830 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.087297 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.087904 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.088742 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.088937 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.089029 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.091673 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:28.091675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:28.091675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:28.091681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:28.093360 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.093719 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.093720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.093720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.093773 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.093990 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.093991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.093991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.094143 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:28.094343 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:28.094381 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 11:22:28.094458 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:22:28.097553 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.090498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38345 (local address 127.0.0.1:40078)
0927 11:22:28.090593 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:28.090603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:28.090646 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:28.090806 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:28.090806 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:28.090913 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:28.090943 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:28.090944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.091829 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.091830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:28.092432 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:28.092434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.092831 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.092831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:28.092880 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.093323 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:28.093328 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:28.094795 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:28.096443 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:28.096443 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:28.096444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:28.096604 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:28.096830 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:28.096830 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:28.096831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:28.096889 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:28.097182 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:28.097183 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:28.097233 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 11:22:28.097320 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 11:22:28.097437 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 11:22:28.097563 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.090556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.0.98.60:38345)
0927 11:22:28.090708 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.090710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.090715 (+ 5us) server_negotiation.cc:407] Connection header received
0927 11:22:28.090730 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.090730 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.090739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.090786 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.091019 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.091747 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.092517 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.092747 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.092828 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.094884 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:28.094885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:28.094885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:28.094892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:28.096306 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.096690 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.096690 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.096691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.096740 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.096950 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.096951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.096951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.097110 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:28.097184 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:28.097381 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 11:22:28.097483 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":139}
Sep 27 11:22:28 dist-test-slave-lsvg krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:22:28.099861 23234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589: Bootstrap starting.
I20260927 11:22:28.100864 23234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:28.101238 23234 log.cc:824] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:28.101431 23084 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } attempt: 2
I20260927 11:22:28.102157 23234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589: No bootstrap required, opened a new log
I20260927 11:22:28.102967 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.095272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.0.98.61:46131)
0927 11:22:28.095335 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.095337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.095345 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:22:28.095369 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.095370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.095382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.095447 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.096065 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.096544 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.097310 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.097544 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.097634 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.100198 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:28.100199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:28.100200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:28.100207 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:28.101812 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.102181 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.102182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.102182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.102238 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.102449 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.102450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.102451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.102570 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:28.102740 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:28.102785 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 11:22:28.102862 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:22:28.104444 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:22:28.104415 23234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } }
I20260927 11:22:28.104580 23234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c63758a96f4a44b074f064f949f589, State: Initialized, Role: FOLLOWER
I20260927 11:22:28.104689 23234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } }
I20260927 11:22:28.105026 23240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } } }
I20260927 11:22:28.105149 23240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:22:28.105222 23234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589 [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 11:22:28 dist-test-slave-lsvg krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:22:28.105415 23058 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cef535dad7d0467d9ca6371c66e5bb78"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37424
W20260927 11:22:28.107391 23251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:22:28.107450 23251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:22:28.109652 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.101701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38345 (local address 127.0.0.1:40082)
0927 11:22:28.101771 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:28.101783 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:28.101827 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:28.102022 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:28.102024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:28.102189 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:28.102241 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:28.102244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.103145 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.103146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:28.103798 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:28.103800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.104258 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.104259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:28.104329 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.104721 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:28.104726 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:28.106579 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:28.108394 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:28.108395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:28.108396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:28.108619 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:28.108854 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:28.108855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:28.108855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:28.108921 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:28.109334 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:28.109336 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:28.109397 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 11:22:28.109459 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 11:22:28.109557 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 11:22:28.110110 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.101732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40082 (local address 127.0.98.60:38345)
0927 11:22:28.101895 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.101897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.101904 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:22:28.101924 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.101925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.101933 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.102324 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.102356 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.103051 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.103904 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.104196 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.104260 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.106658 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:28.106660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:28.106660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:28.106668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:28.108300 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.108699 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.108700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.108700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.108770 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.108993 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.108994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.108994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.109220 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:28.109299 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:28.109948 (+ 649us) server_negotiation.cc:299] Negotiation successful
0927 11:22:28.110039 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260927 11:22:28.107419 23166 master_runner.cc:186] Error getting master registration for 127.0.98.62:43417: OK, instance_id { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" instance_seqno: 1790508147758000 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 11:22:28.110441 23166 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 11:22:28.111599 23084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78: Bootstrap starting.
I20260927 11:22:28.112354 23084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:28.112681 23084 log.cc:824] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:28.113567 23084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78: No bootstrap required, opened a new log
I20260927 11:22:28.115000 23084 raft_consensus.cc:348] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } }
I20260927 11:22:28.115156 23084 raft_consensus.cc:729] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef535dad7d0467d9ca6371c66e5bb78, State: Initialized, Role: FOLLOWER
I20260927 11:22:28.115279 23084 consensus_queue.cc:260] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } }
I20260927 11:22:28.115625 23084 sys_catalog.cc:564] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:28.115674 23058 raft_consensus.cc:482] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 11:22:28.115651 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } } }
I20260927 11:22:28.115741 23058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:28.115751 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:22:28.116510 23058 raft_consensus.cc:504] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } }
I20260927 11:22:28.116743 23058 leader_election.cc:288] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [CANDIDATE]: Term 1 election: Requested vote from peers dc69efcbddef4ed387f79e44e33bbeb0 (127.0.98.61:46131), 43c63758a96f4a44b074f064f949f589 (127.0.98.60:38345)
I20260927 11:22:28.117069 23208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cef535dad7d0467d9ca6371c66e5bb78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "43c63758a96f4a44b074f064f949f589"
I20260927 11:22:28.117188 23208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:28.118216 23208 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cef535dad7d0467d9ca6371c66e5bb78 in term 1.
I20260927 11:22:28.118863 23035 leader_election.cc:302] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c63758a96f4a44b074f064f949f589, cef535dad7d0467d9ca6371c66e5bb78; no voters:
I20260927 11:22:28.119026 23254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:22:28.119226 23254 raft_consensus.cc:686] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [term 1 LEADER]: Becoming Leader. State: Replica: cef535dad7d0467d9ca6371c66e5bb78, State: Running, Role: LEADER
I20260927 11:22:28.119343 23254 consensus_queue.cc:237] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } }
W20260927 11:22:28.119556 23265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:22:28.119694 23265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:22:28.119762 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cef535dad7d0467d9ca6371c66e5bb78. Latest consensus state: current_term: 1 leader_uuid: "cef535dad7d0467d9ca6371c66e5bb78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } } }
I20260927 11:22:28.119846 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:28.120982 23016 master_runner.cc:186] Error getting master registration for 127.0.98.61:46131: OK, instance_id { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" instance_seqno: 1790508147907677 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 11:22:28.123345 23016 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 11:22:28.125897 23276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:22:28.126488 23276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:22:28.127426 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.118031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40084 (local address 127.0.98.60:38345)
0927 11:22:28.118073 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.118075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.119345 (+ 1270us) server_negotiation.cc:407] Connection header received
0927 11:22:28.119534 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.119535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.119548 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.119590 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.120481 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.121289 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.122321 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.122624 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.122666 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.124323 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:28.124325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:28.124325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:28.124333 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:28.126028 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.126486 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.126487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.126488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.126543 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.126727 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.126728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.126728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.126851 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:28.126938 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:28.127252 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 11:22:28.127332 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:22:28.130050 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.118428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.0.98.62:43417)
0927 11:22:28.118499 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.118514 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.120090 (+ 1576us) server_negotiation.cc:407] Connection header received
0927 11:22:28.120255 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.120256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.120279 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.120308 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.121487 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.122211 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.125179 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.125474 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.125517 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.127219 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:28.127221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:28.127221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:28.127229 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:28.128704 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.129190 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.129191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.129191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.129247 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.129456 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.129457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.129457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.129578 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:28.129650 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:28.129867 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 11:22:28.129942 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 11:22:28.131067 23208 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589 [term 1 FOLLOWER]: Refusing update from remote peer cef535dad7d0467d9ca6371c66e5bb78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:22:28.131430 23274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:22:28.133312 23240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cef535dad7d0467d9ca6371c66e5bb78. Latest consensus state: current_term: 1 leader_uuid: "cef535dad7d0467d9ca6371c66e5bb78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } } }
I20260927 11:22:28.133412 23240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:22:28.139137 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.118641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.0.98.61:46131)
0927 11:22:28.118695 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.118698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.122352 (+ 3654us) server_negotiation.cc:407] Connection header received
0927 11:22:28.122605 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.122606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.122630 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.122662 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.125150 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.126362 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.128382 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.128691 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.128769 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.131151 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:28.131152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:28.131152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:28.131160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:28.132880 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.137804 (+ 4924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.137807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.137807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.137907 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.138292 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.138292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.138293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.138458 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:28.138561 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:28.138905 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 11:22:28.139010 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 11:22:28.144665 23159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } attempt: 3
I20260927 11:22:28.147112 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cef535dad7d0467d9ca6371c66e5bb78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } } }
I20260927 11:22:28.147236 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:28.147802 23240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cef535dad7d0467d9ca6371c66e5bb78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } } }
I20260927 11:22:28.147897 23240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:22:28.148041 23278 mvcc.cc:205] Tried to move back new op lower bound from 7333921374741250048 to 7333921374697525248. Current Snapshot: MvccSnapshot[applied={T|T < 7333921374741250048}]
I20260927 11:22:28.148353 23277 mvcc.cc:205] Tried to move back new op lower bound from 7333921374741250048 to 7333921374697525248. Current Snapshot: MvccSnapshot[applied={T|T < 7333921374741250048}]
Sep 27 11:22:28 dist-test-slave-lsvg krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790508147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:22:28.150652 23276 catalog_manager.cc:1347] Generated new cluster ID: 0b6f89fe1478424fbb933ea619a309de
I20260927 11:22:28.150703 23276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:22:28.153854 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.144984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38345 (local address 127.0.0.1:40096)
0927 11:22:28.145036 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 11:22:28.145078 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 11:22:28.145120 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:22:28.145331 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:22:28.145332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:22:28.145534 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:22:28.145586 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:28.145589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.146567 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.146568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:28.147426 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:22:28.147429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.148183 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.148184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:22:28.148244 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.149418 (+ 1174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:22:28.149423 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:22:28.151025 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:22:28.152751 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:28.152753 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:28.152753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:28.152969 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:28.153193 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:22:28.153194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:22:28.153194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:22:28.153258 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:22:28.153537 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:22:28.153538 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:22:28.153592 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 11:22:28.153646 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 11:22:28.153746 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 11:22:28.154377 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:28.145128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.0.98.60:38345)
0927 11:22:28.145196 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.145198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.145205 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:22:28.145229 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.145230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.145240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.145287 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.145667 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.146489 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.147676 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:28.147992 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:28.148055 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:28.151115 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:22:28.151116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:22:28.151117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:22:28.151124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:22:28.152654 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.153040 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.153041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.153042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.153124 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:22:28.153323 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:22:28.153324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:22:28.153324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:22:28.153457 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:22:28.153506 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:28.154220 (+ 714us) server_negotiation.cc:299] Negotiation successful
0927 11:22:28.154294 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:22:28.156023 23159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0: Bootstrap starting.
I20260927 11:22:28.156823 23159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0: Neither blocks nor log segments found. Creating new log.
I20260927 11:22:28.157172 23159 log.cc:824] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0: Log is configured to *not* fsync() on all Append() calls
I20260927 11:22:28.158162 23159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0: No bootstrap required, opened a new log
I20260927 11:22:28.160218 23159 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } }
I20260927 11:22:28.160378 23159 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc69efcbddef4ed387f79e44e33bbeb0, State: Initialized, Role: FOLLOWER
I20260927 11:22:28.160467 23159 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } }
I20260927 11:22:28.160889 23133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cef535dad7d0467d9ca6371c66e5bb78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dc69efcbddef4ed387f79e44e33bbeb0"
I20260927 11:22:28.160977 23132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:22:28.160873 23159 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:22:28.161880 23132 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0 [term 1 FOLLOWER]: Refusing update from remote peer cef535dad7d0467d9ca6371c66e5bb78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:22:28.162314 23254 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:22:28.163714 23133 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cef535dad7d0467d9ca6371c66e5bb78 in term 1.
W20260927 11:22:28.164479 23294 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:22:28.164535 23294 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:22:28.165484 23297 mvcc.cc:205] Tried to move back new op lower bound from 7333921374741250048 to 7333921374697525248. Current Snapshot: MvccSnapshot[applied={T|T < 7333921374741250048}]
I20260927 11:22:28.160808 23283 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } } }
I20260927 11:22:28.165843 23283 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:22:28.165947 23283 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cef535dad7d0467d9ca6371c66e5bb78. Latest consensus state: current_term: 1 leader_uuid: "cef535dad7d0467d9ca6371c66e5bb78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } } }
I20260927 11:22:28.166064 23283 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:22:28.166139 23283 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cef535dad7d0467d9ca6371c66e5bb78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } } }
I20260927 11:22:28.166332 23283 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:22:28.167325 23276 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:22:28.168169 23276 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:22:28.173214 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cef535dad7d0467d9ca6371c66e5bb78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef535dad7d0467d9ca6371c66e5bb78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43417 } } peers { permanent_uuid: "dc69efcbddef4ed387f79e44e33bbeb0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46131 } } peers { permanent_uuid: "43c63758a96f4a44b074f064f949f589" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38345 } } }
I20260927 11:22:28.173336 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78 [sys.catalog]: This master's current role is: LEADER
I20260927 11:22:28.177237 23276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cef535dad7d0467d9ca6371c66e5bb78: Generated new TSK 0
I20260927 11:22:28.177455 23276 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 11:22:28.206053 23238 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:28.203253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.0.98.60:38345)
0927 11:22:28.203319 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.203322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.205232 (+ 1910us) server_negotiation.cc:407] Connection header received
0927 11:22:28.205369 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.205370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.205383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.205417 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.205972 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40110: BlockingRecv error: recv got EOF from 127.0.0.1:40110 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 11:22:28.206959 23087 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:28.203754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.0.98.62:43417)
0927 11:22:28.203801 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.203803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.206380 (+ 2577us) server_negotiation.cc:407] Connection header received
0927 11:22:28.206489 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.206491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.206503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.206536 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.206885 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37436: BlockingRecv error: recv got EOF from 127.0.0.1:37436 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 11:22:28.207782 23163 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:28.204097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.0.98.61:46131)
0927 11:22:28.204142 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.204144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.207214 (+ 3070us) server_negotiation.cc:407] Connection header received
0927 11:22:28.207340 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.207342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.207354 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.207386 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.207712 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43398: BlockingRecv error: recv got EOF from 127.0.0.1:43398 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 11:22:28.214391 23163 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:28.213416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.0.98.61:46131)
0927 11:22:28.213477 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.213479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.213731 (+ 252us) server_negotiation.cc:407] Connection header received
0927 11:22:28.213865 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.213866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.213890 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.213922 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.214330 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43412: BlockingRecv error: recv got EOF from 127.0.0.1:43412 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 11:22:28.218295 23163 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:28.217343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.0.98.61:46131)
0927 11:22:28.217390 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.217392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.217670 (+ 278us) server_negotiation.cc:407] Connection header received
0927 11:22:28.217784 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.217785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.217797 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.217830 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.218224 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43428: BlockingRecv error: recv got EOF from 127.0.0.1:43428 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 11:22:28.222622 23163 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:28.221655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.0.98.61:46131)
0927 11:22:28.221697 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.221699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.221990 (+ 291us) server_negotiation.cc:407] Connection header received
0927 11:22:28.222110 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.222111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.222124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.222156 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.222544 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43430: BlockingRecv error: recv got EOF from 127.0.0.1:43430 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 11:22:28.228952 23163 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:28.227928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.0.98.61:46131)
0927 11:22:28.227972 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.227974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.228290 (+ 316us) server_negotiation.cc:407] Connection header received
0927 11:22:28.228408 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.228409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.228422 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.228456 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.228888 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43432: BlockingRecv error: recv got EOF from 127.0.0.1:43432 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 11:22:28.239529 23163 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:28.238441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.0.98.61:46131)
0927 11:22:28.238491 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.238494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.238815 (+ 321us) server_negotiation.cc:407] Connection header received
0927 11:22:28.238928 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.238929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.238943 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.238977 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.239449 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43444: BlockingRecv error: recv got EOF from 127.0.0.1:43444 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 11:22:28.258354 23163 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:28.257260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.0.98.61:46131)
0927 11:22:28.257323 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.257326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.257564 (+ 238us) server_negotiation.cc:407] Connection header received
0927 11:22:28.257693 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.257695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.257708 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.257744 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.258286 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43460: BlockingRecv error: recv got EOF from 127.0.0.1:43460 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 11:22:28.293260 23163 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:28.291963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.0.98.61:46131)
0927 11:22:28.292018 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.292020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.292379 (+ 359us) server_negotiation.cc:407] Connection header received
0927 11:22:28.292505 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.292506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.292520 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.292555 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.293194 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43470: BlockingRecv error: recv got EOF from 127.0.0.1:43470 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 11:22:28.359642 23163 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:28.358762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43486 (local address 127.0.98.61:46131)
0927 11:22:28.358854 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.358856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.358948 (+ 92us) server_negotiation.cc:407] Connection header received
0927 11:22:28.358981 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.358982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.358994 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.359064 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.359535 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43486: BlockingRecv error: recv got EOF from 127.0.0.1:43486 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 11:22:28.489745 23163 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:28.488883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.0.98.61:46131)
0927 11:22:28.488978 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.488980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.489096 (+ 116us) server_negotiation.cc:407] Connection header received
0927 11:22:28.489195 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.489196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.489208 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.489303 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.489680 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43498: BlockingRecv error: recv got EOF from 127.0.0.1:43498 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 11:22:28.747527 23163 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:22:28.746932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.0.98.61:46131)
0927 11:22:28.746990 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:28.746996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:28.747006 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:22:28.747056 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:28.747057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:28.747068 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:28.747131 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:22:28.747446 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43502: BlockingRecv error: recv got EOF from 127.0.0.1:43502 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 11:22:29.107928 23251 catalog_manager.cc:1259] Loaded cluster ID: 0b6f89fe1478424fbb933ea619a309de
I20260927 11:22:29.108016 23251 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589: loading cluster ID for follower catalog manager: success
I20260927 11:22:29.108862 23251 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589: acquiring CA information for follower catalog manager: success
I20260927 11:22:29.109237 23251 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43c63758a96f4a44b074f064f949f589: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 11:22:29.164975 23294 catalog_manager.cc:1259] Loaded cluster ID: 0b6f89fe1478424fbb933ea619a309de
I20260927 11:22:29.165096 23294 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0: loading cluster ID for follower catalog manager: success
I20260927 11:22:29.165961 23294 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0: acquiring CA information for follower catalog manager: success
I20260927 11:22:29.166344 23294 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dc69efcbddef4ed387f79e44e33bbeb0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 11:22:29.263634 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:29.260830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.0.98.61:46131)
0927 11:22:29.260975 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:29.260977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:29.260987 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:22:29.261083 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:29.261084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:29.261152 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:29.261260 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:29.261446 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:29.261941 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:29.262855 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:29.263054 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:29.263134 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:29.263309 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:29.263419 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:29.263480 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:29.263513 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 11:22:29.263533 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260927 11:22:29.267778 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:29.265478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.0.98.60:38345)
0927 11:22:29.265574 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:29.265575 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:29.265583 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:22:29.265623 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:29.265624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:29.265657 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:29.265731 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:29.265811 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:29.266268 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:29.266996 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:29.267178 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:29.267226 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:29.267375 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:29.267471 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:29.267528 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:29.267653 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 11:22:29.267670 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":41,"threads_started":1}
I20260927 11:22:29.272535 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:29.269675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.0.98.61:46131)
0927 11:22:29.269723 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:29.269745 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:29.269904 (+ 159us) server_negotiation.cc:407] Connection header received
0927 11:22:29.270028 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:29.270029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:29.270041 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:29.270088 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:29.270228 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:29.270691 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:29.271771 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:29.272073 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:29.272139 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:29.272281 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:29.272320 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:29.272371 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:29.272425 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 11:22:29.272450 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:22:29.272604 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:29.269624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40134 (local address 127.0.98.60:38345)
0927 11:22:29.269665 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:29.269667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:29.269676 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:22:29.269988 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:29.269990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:29.270000 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:29.270070 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:29.270192 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:29.270921 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:29.271904 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:29.272147 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:29.272212 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:29.272324 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:29.272357 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:29.272403 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:29.272507 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 11:22:29.272526 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:22:29.272926 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:22:29.269675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.0.98.62:43417)
0927 11:22:29.269787 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 11:22:29.269789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:22:29.269844 (+ 55us) server_negotiation.cc:407] Connection header received
0927 11:22:29.269930 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:22:29.269931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:22:29.269996 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:22:29.270075 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:22:29.270254 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:29.270993 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:29.271717 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:22:29.272007 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:22:29.272068 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:22:29.272286 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:22:29.272421 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:22:29.272475 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 11:22:29.272629 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 11:22:29.272708 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":60,"threads_started":1}
I20260927 11:22:29.279844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 23016
I20260927 11:22:29.285674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 23091
I20260927 11:22:29.290567 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuJOgQn/build/release/bin/kudu with pid 23166
2026-09-27T11:22:29Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2826 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5696 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271149 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST